Word and Sentence Tokenization Command-Line Trainer

Tokenization, splitting text into words and sentences, is the very first operation in every NLP pipeline, and it is deceptively subtle: abbreviations break naive sentence splitting, contractions and hyphenation complicate word boundaries, and different libraries resolve these cases differently. Anyone building on top of NLP tools needs to know what their tokenizer actually does. The intern builds a Python command-line application that tokenizes input text into words and sentences using both NLTK and spaCy. The tool accepts text directly or from files, runs both tokenizers, and presents the results clearly, showing token counts, sentence boundaries, and a comparison view highlighting where the two libraries split the same passage differently. The intern feeds it tricky inputs, abbreviations like Dr., decimal numbers, quotations, and URLs, to surface exactly the edge cases that make tokenization an interesting engineering problem rather than a string split. Through this focused project the intern learns the foundational operation underlying all text processing, gains first hands-on experience with the two dominant Python NLP libraries, and practices building a clean command-line interface with helpful output formatting, producing a small utility that doubles as a study aid for every NLP course exercise that follows.