Effective writing style transfer via combinatorial paraphrasing

Tommi Gröndahl 1  and N. Asokan 2
  • 1 Aalto University, , Konemiehentie 2, 02150, Espoo, Finland
  • 2 University of Waterloo, , 200 University Ave W, Waterloo, Canada

Abstract

Stylometry can be used to profile or deanonymize authors against their will based on writing style. Style transfer provides a defence. Current techniques typically use either encoder-decoder architectures or rule-based algorithms. Crucially, style transfer must reliably retain original semantic content to be actually deployable. We conduct a multifaceted evaluation of three state-of-the-art encoder-decoder style transfer techniques, and show that all fail at semantic retainment. In particular, they do not produce appropriate paraphrases, but only retain original content in the trivial case of exactly reproducing the text. To mitigate this problem we propose ParChoice: a technique based on the combinatorial application of multiple paraphrasing algorithms. ParChoice strongly outperforms the encoder-decoder baselines in semantic retainment. Additionally, compared to baselines that achieve nonnegligible semantic retainment, ParChoice has superior style transfer performance. We also apply ParChoice to multi-author style imitation (not considered by prior work), where we achieve up to 75% imitation success among five authors. Furthermore, when compared to two state-of-the-art rule-based style transfer techniques, ParChoice has markedly better semantic retainment. Combining ParChoice with the best performing rulebased baseline (Mutant-X [34]) also reaches the highest style transfer success on the Brennan-Greenstadt and Extended-Brennan-Greenstadt corpora, with much less impact on original meaning than when using the rulebased baseline techniques alone. Finally, we highlight a critical problem that afflicts all current style transfer techniques: the adversary can use the same technique for thwarting style transfer via adversarial training. We show that adding randomness to style transfer helps to mitigate the effectiveness of adversarial training.

If the inline PDF is not rendering correctly, you can download the PDF file here.

  • [1] Ahmed Abbasi and Hsinchun Chen. Writeprints: A stylometric approach to identity-level identification and similarity detection in cyberspace. ACM Transactions on Information and System Security, 26(2):1–29, 2008.

  • [2] Sadia Afroz, Michael Brennan, and Rachel Greenstadt. Detecting Hoaxes, Frauds, and Deception in Writing Style Online. In Proceedings of the 2012 IEEE Symposium on Security and Privacy, pages 461–475, 2012.

  • [3] Mishari Almishari, Ekin Oguz, and Gene Tsudik. Fighting Authorship Linkability with Crowdsourcing. In Proceedings of the second ACM conference on Online social networks, pages 69–82, 2014.

  • [4] Douglas Bagnall. Author identification using multi-headed recurrent neural networks. In Working Notes of CLEF 2015 - Conference and Labs of the Evaluation Forum, 2015.

  • [5] Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, 2005.

  • [6] Steven Bird, Ewan Klein, and Edward Loper. Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit. O’Reilly, Beijing, 2009.

  • [7] Michael Brennan, Sadia Afroz, and Rachel Greenstadt. Adversarial stylometry: Circumventing authorship recognition to preserve privacy and anonymity. ACM Transactions on Information and System Security, 15(3):12:1–12:22, 2011.

  • [8] Michael Brennan and Rachel Greenstadt. Practical Attacks Against Authorship Recognition Techniques. In Proceedings of the Twenty-First Conference on Innovative Applications of Artificial Intelligence, pages 60–65, 2009.

  • [9] Marcelo Luiz Brocardo, Issa Traore, Isaac Woungang, and Mohammad S. Obaidat. Authorship verification using deep belief network systems. Communication Systems, 30(12), 2017.

  • [10] Aylin Caliskan and Rachel Greenstadt. Translate once, translate twice, translate thrice and attribute: Identifying authors and machine translation tools in translated text. In Proceedings of the 2012 IEEE Sixth International Conference on Semantic Computing (ICSC), pages 121–125, 2012.

  • [11] Andrew Carnie. Constituent Structure. Oxford University Press, Oxford, 2008.

  • [12] Daniel Castro-Castro, Reynier Ortega Bueno, and Rafael Mu noz. Author masking by sentence transformation – notebook for PAN at CLEF. In Working notes of CLEF2017, 2017.

  • [13] C.K. Chow. On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory, 16:41–46, 1970.

  • [14] Siobahn Day, James Brown, Zachery Thomas, India Gregory, Lowell Bass, and Gerry Dozier. Adversarial Authorship, AuthorWebs, and entropy-based evolutionary clustering. In Proceedings of the 25th International Conference on Computer Communication and Networks (ICCCN), pages 1–6, 2016.

  • [15] Iqbal Farkhund, Hamad Binsalleeh, Benjamin C.M. Fung, and Mourad Debbabi. Mining writeprints from anonymous e-mails for forensic investigation. Digital Investigation, 7(1–2):56–64, 2013.

  • [16] César Ferri, Peter Flach, and José Hernández-Orallo. Delegating classifiers. In Proceedings of the 21th International Conference on Machine Leaning (ICML’04), pages 106–110, 2004.

  • [17] Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. Style transfer in text: Exploration and evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 663–670, 2018.

  • [18] Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. PPDB: The paraphrase database. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pages 758–764, 2013.

  • [19] Zhenhao Ge and Yufang Sun. Domain specific author attribution based on feedforward neural network language models. In Proceedings of the 5th International Conference on Pattern Recognition Applications and Methods (ICPRAM), pages 597–604, 2016.

  • [20] David Goldberg. Genetic Algorithms in Search, Optimization and Machine Learning. Addison-Wesley Longman Publishing Co., Inc., Boston, 1989.

  • [21] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In Proceedings of the International Conference on Neural Information Processing Systems (NIPS), pages 2672–2680, 2014.

  • [22] Jack Grieve. Quantitative authorship attribution: An evaluation of techniques. Literary and Linguistic Computing, 22(3):251–270, 2007.

  • [23] Tommi Gröndahl and N. Asokan. EAT2seq: A generic framework for controlled sentence transformation without taskspecific training. arXiv preprint arXiv: 1902.09381, 2019.

  • [24] Tommi Gröndahl and N. Asokan. Text analysis in adversarial settings: Does deception leave a stylistic trace? ACM Computing Surveys, 52(3):45:1–45:36, 2019.

  • [25] Tommi Gröndahl, Luca Pajola, Mika Juuti, Mauro Conti, and N. Asokan. All you need is “love”: Evading hate speech detection. In Proceedings of the 11th ACM Workshop on Artificial Intelligence and Security (AISec’11), pages 2–12, 2018.

  • [26] Michael Haardt. GNU Diction, 2005 (accessed February 24, 2020). https://www.gnu.org/software/diction/.

  • [27] Thanh Nghia Ho and Wee Keong Ng. Application of stylometry to DarkWeb forum user identification. In Proceedings of Information and Communications Security, pages 173–183, 2016.

  • [28] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997.

  • [29] Charles D. Hollingsworth. Syntactic Stylometry: Using Sentence Structure for Authorship Attribution. PhD thesis, University of Georgia, 2012.

  • [30] Charles D. Hollingsworth. Using dependency-cased annotations for authorship identification. In Proceedings of the International Conference on Text, Speech and Dialogue, pages 314–319, 2012.

  • [31] Matthew Honnibal and Mark Johnson. An improved nonmonotonic transition system for dependency parsing. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP), pages 1373–1378, 2015.

  • [32] Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. Toward controlled generation of text. In Proceedings of the 34th International Conference on Machine Learning, 2017.

  • [33] Patrick Juola. Large-scale experiments in authorship attribution. English Studies, 93(3):275–283, 2012.

  • [34] Patrick Juola. Stylometry and immigration: A case study. Journal of Law and Policy, 21(2):287–298, 2013.

  • [35] Mika Juuti, Bo Sun, Tatsuya Mori, and N. Asokan. Stay ontopic: Generating context-specific fake restaurant reviews. In Proceedings of the 23rd European Symposium on Research in Computer Security (ESORICS), pages 132–151, 2018.

  • [36] Georgi Karadzhov, Tsvetomila Mihaylova, Yasen Kiprov, Georgi Georgiev, Ivan Koychev, and Preslav Nakov. The case for being average: A mediocrity approach to style masking and author obfuscation. In Proceedings of the International Conference of the Cross-Language Evaluation Forum for European Languages (CLEF), pages 173–185, 2017.

  • [37] Foaad Khosmood. Comparison of sentence-level paraphrasing approaches for statistical style transformation. In Proceedings of the International Conference on Artificial Intelligence, 2012.

  • [38] Foaad Khosmood and Robert Levinson. Automatic natural language style classification and transformation. In Proceedings of the 2008 BCS-IRSG Conference on Corpus Profiling, page 3, 2008.

  • [39] Foaad Khosmood and Robert Levinson. Toward automated stylistic transformation of natural language text. In Proceedings of the Digital Humanities, pages 177–181, 2009.

  • [40] Foaad Khosmood and Robert Levinson. Automatic synonym and phrase replacement show promise for style transformation. In Proceedings of The Ninth International Conference on Machine Learning and Applications, pages 958–961, 2010.

  • [41] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, pages 2278–2324, 1998.

  • [42] Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. Textbugger: Generating adversarial text againstreal-world applications. In Proceedings of Network and Distributed Systems Security (NDSS), 2019.

  • [43] Lajanugen Logeswaran, Honglak Lee, and Samy Bengio. Content preserving text generation with attribute controls. In Proceedings of the International Conference on Neural Information Processing Systems (NIPS), 2018.

  • [44] Nathan Mack, Jasmine Bowers, Henry Williams, Gerry Dozier, and Joseph Shelton. The best way to a strong defense is a strong offense: Mitigating deanonymization attacks via iterative language translation. International Journal of Machine Learning and Computing, 5(5):409–413, 2015.

  • [45] Asad Mahmood, Faizan Ahmad, Zubair Shafiq, Padmini Srinivasan, and Fareed Zaffar. A girl has no name: Automated authorship obfuscation using Mutant-X. In Proceedings on Privacy Enhancing Technologies (PETS), pages 54–71, 2019.

  • [46] Muharram Mansoorizadeh, Taher Rahgooy, Mohammad Aminiyan, and Mahdy Eskandari. Author obfuscation using WordNet and language models – notebook for PAN at CLEF 2016. In CLEF 2016 Evaluation Labs and Workshop – Working Notes Papers, 2016.

  • [47] Andrew W.E. McDonald, Sadia Afroz, Aylin Caliskan, Ariel Stolerman, and Rachel Greenstadt. Use fewer instances of the letter i: Toward writing style anonymization. In Simone Fischer-Hübner and Matthew Wright, editors, Privacy Enhancing Technologies. Volume 7384 of Lecture Notes in Computer Science, pages 299–318. 2012.

  • [48] Tsvetomila Mihaylova, Georgi Karadjov, Preslav Nakov, Yasen Kiprov, Georgi Georgiev, and Ivan Koychev. SU@PAN’2016: Author obfuscation – notebook for PAN at CLEF 2016. In CLEF 2016 Evaluation Labs and Workshop – Working Notes Papers, 2016.

  • [49] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In Proceedings of the International Conference on Neural Information Processing Systems (NIPS), page 3111–3119, 2013.

  • [50] George A. Miller. WordNet: A lexical database for English. Communications of the ACM, 38(11):39–41, 1995.

  • [51] Arvind Narayanan, Hristo Paskov, Neil Zhenqiang Gong, John Bethencourt, Emil Stefanov, Eui Chul Richard Shin, and Dawn Song. On the feasibility of internet-scale author identification. In Proceedings of the IEEE Symposium on Security and Privacy, pages 300–314, 2012.

  • [52] Roberto Navigli. Word sense disambiguation: A survey. ACM Computing Surveys, 41(2):1–69, 2009.

  • [53] Rebekah Overdorf and Rachel Greenstadt. Blogs, twitter feeds, and reddit comments: Cross-domain authorship attribution. In Proceedings on Privacy Enhancing Technologies (PETS), pages 155–171, 2016.

  • [54] Lluís Padró and Evgeny Stanilovsky. Freeling 3.0: Towards wider multilinguality. In Proceedings of the Language Resources and Evaluation Conference (LREC 2012), pages 2473—-479, 2012.

  • [55] Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Short Papers), pages 425–430, 2015.

  • [56] Martin Potthast, Sarah Braun, Tolga Buz, Fabian Duffhauss, Florian Friedrich, Jörg Marvin Gülzow, Jakob Köhler, Winfried Lötzsch, Fabian Müller, Maike Elisa Müller, Robert Paßmann, Bernhard Reinke, Lucas Rettenmeier, Thomas Rometsch, Timo Sommer, Michael Träger, Sebastian Wilhelm, Benno Stein, Efstathios Stamatatos, and Matthias Hagen. Who wrote the web? Revisiting influential author identification research applicable to information retrieval. In Nicola Ferro, Fabio Crestani, Marie-Francine Moens, Josiane Mothe, Fabrizio Silvestri, Giorgio Maria Di Nunzio, Claudia Hauff, and Gianmaria Silvello, editors, Advances in Information Retrieval, pages 393–407. Springer International Publishing, 2016.

  • [57] Martin Potthast, Matthias Hagen, and Benno Stein. Author obfuscation: Attacking the state of the art in authorship verification. In CLEF 2016 Working Notes, 2016.

  • [58] Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W. Black. Style Transfer Through Back-Translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pages 866–876, 2018.

  • [59] Daniel Preotiuc-Pietro, Jordan Carpenter, and Lyle Ungar. Personality driven differences in paraphrase preference. In Proceedings of the Second Workshop on Natural Language Processing and Computational Social Science, pages 17–26, 2017.

  • [60] Press Freedom Index. RSF Index 2018: Hatred of journalism threatens democracies. https://rsf.org/en/rsf-index-2018-hatred-journalism-threatens-democracies (May 1st 2018).

  • [61] Sravana Reddy and Kevin Knight. Obfuscating gender in social media writing. In Proceedings of Workshop on Natural Language Processing and Computational Social Science, pages 17–26, 2016.

  • [62] Jonathan Schler, Moshe Koppel, Shlomo Argamon, and James Pennebaker. Effects of age and gender on blogging. In Proceedings of AAAI Spring Symposium on Computational Approaches for Analyzing Weblogs, 2006.

  • [63] Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. Style transfer from non-parallel text by crossalignment. In Proceedings of the International Conference on Neural Information Processing Systems (NIPS), 2017.

  • [64] Rakshith Shetty, Bernt Schiele, and Mario Fritz. A4NT: Author attribute anonymity by adversarial training of neural machine translation. In Proceedings of the 27th USENIX Security Symposium, pages 1633–1650, 2018.

  • [65] Efstathios Stamatatos. A survey of modern authorship attribution methods. Journal of the American Society for Information Science and Technology, 60(3):538–556, 2009.

  • [66] Ariel Stolerman, Rebekah Overdorf, Sadia Afroz, and Rachel Greenstadt. Breaking the closed-world assumption in stylometric authorship attribution. In Gilbert Peterson and Sujeet Shenoi, editors, Advances in Digital Forensics X, pages 185–205, Berlin, Heidelberg, 2014. Springer Berlin Heidelberg.

  • [67] Kl Surendran, O.P. Harilal, Hrudya Poroli, Prabaharan Poornachandran, and N.K. Suchetha. Stylometry detection using deep learning. In Proceedings of Computational Intelligence in Data Mining, pages 749–757, 2017.

  • [68] Neal Tempestt, Kalaivani Sundararajan, Aneez Fatima, Yiming Yan, Yingfei Xiang, and Damon Woodard. Surveying stylometry techniques and applications. ACM Computing Surveys, 50(6), 2017.

  • [69] Louis Tesnière. Èléments de syntaxe structurale. Klincksieck, Paris, 1959.

  • [70] Jingjing Xu, Xu Sun, Qi Zeng, Xuancheng Ren, Xiaodong Zhang, Houfeng Wang, and Wenjie Li. Unpaired sentiment-to-sentiment translation: A cycled reinforcement learning approach. In Proceedings of the Association for Computational Linguistics (ACL), pages 979–988, 2018.

  • [71] Zichao Yang, Zhiting Hu, Chris Dyer, Eric P Xing, and Taylor Berg-Kirkpatrick. Unsupervised text style transfer using language models as discriminators. In Proceedings of the International Conference on Neural Information Processing (NIPS), pages 7287–7298, 2018.

  • [72] Liang-Chih Yu, Jin Wang, K. Robert Lai, and Xuejie Zhang. Refining word embeddings using intensity scores for sentiment analysis. In IEEE/ACM Transactions on Audio, Speech, and Language Processing, pages 671–681, 2018.

  • [73] Jake Zhao, Yoon Kim, Kelly Zhang, Alexander M. Rush, and Yann LeCun. Adversarially regularized autoencoders. In Proceedings of the 35th International Conference on Machine Learning, pages 5902–5911, 2018.

  • [74] Rong Zheng, Jiexun Li, Hsinchun Chen, and Zan Huang. A framework of authorship identification for online messages: Writing style features and classification techniques. Journal of the American Society for Information Science and Technology, 57(3):378–393, 2006.

OPEN ACCESS

Journal + Issues

Search