Automatic language classification



#!/usr/bin/python2

import sys
import bz2

def classify(text, langs=('english', 'german', 'french')):
    results = {}
    for lang in langs:
        with open(lang + '.txt') as f:
            corpus = f.read()

        compressed = len(bz2.compress(corpus))
        results[lang] = len(bz2.compress(corpus + text)) - compressed

    return sorted(results, key=results.__getitem__)

if __name__ == '__main__':
    print "Most likely %s." % classify(sys.stdin.read())[0].capitalize()

$ wget -qO - http://www.gutenberg.org/ebooks/31469.txt.utf8 | ./classific.py
Most likely English.
$ wget -qO - http://www.gutenberg.org/ebooks/22367.txt.utf8 | ./classific.py
Most likely German.
$ wget -qO - http://www.gutenberg.org/ebooks/4968.txt.utf8 | ./classific.py
Most likely French.

Fascinating. But I didn't start my own threads whenever I made a shitty program using the language I was learning.

>>1
I see how it works. Compressing an input text of a particular language with a large file containing words also belonging to the same language will result in a smaller compressed size due to pattern aliasing. It's truly ingenious.

Also, >>3 here again, I bet if you were to rigorously analyze the OPs program, one would conclude that it is equivalent to a bag-of-words Bayesian classifier, with the bonus that it's really fucking easy to implement and port to other languages and platforms.

Compressing large files to determinate the language of an input file is not slow enough. The program could at least implement sleepsort.

>>4
equivalent to a bag-of-words Bayesian classifier
Nope, it's even funnier, as bzip2 doesn't care about words, only about frequent sequences of characters. So it doesn't need huge corpora and can correctly recognize texts where none of the words are present in the corpus, also with typos, bad grammar, OCR artifacts etc.

>>5
it's okay, he's using FIOC

http://cairnarvon.rotahall.org/2011/05/10/automatic-language-classification/

XARN QUALITY THREAD

Who the fuck is Xarn?

You can't recognize images this way.

Also, it would fail to recognize voice or other audio data.

>>10
http://dis.4chan.org/read/prog/1307312436

This is Xarn.

>>13
Someone from /prog/ is having sex? Heresy!

>>13
He's not Xarn, idiot.

>>15
You clearly don't understand the Xarnness of Xarn

In a few months, some asshole will publish this to Reddit, finally resulting in a wikipedia article about it. Just wait and see.

>>1
Brilliant!

>>17
/prog/ is so cutting-edge

>>19
This has been around since at least 2002, see ``Language trees and zipping''.

>>20
lol, academy papers are 4chan level posts, but with list sexism


Watch your gendered pronouns. When you write, you'll want to make sure that you don't do anything to make your readers feel excluded. If you use "he" and "him" all the time, you are excluding half of your potential readership. We'll acknowledge that the he/she solution is a bit cumbersome in writing. However, you might solve the problem as we have done in this document: by alternating "he" and "she" throughout. Other writers advocate always using "she" instead of "he" as a way of acknowledging a long-standing exclusion of women from texts. Whatever decision you make in the end, be sensitive to its effect on your readers.

>>21
Watch your gendered pronouns. When you write, you'll want to make sure that you don't do anything to make your readers feel excluded. If you use "he" and "him" all the time, you are excluding half of your potential readership. We'll acknowledge that the he/she solution is a bit cumbersome in writing. However, you might solve the problem as we have done in this document: by alternating "he" and "she" throughout. Other writers advocate always using "she" instead of "he" as a way of acknowledging a long-standing exclusion of women from texts. Whatever decision you make in the end, be sensitive to its effect on your readers.
Nobody gives a shit, fat feminist cunts.

>>21
Using ``she'' instead of ``he'' is stupid and I hate you for doing it.

>>22,23 go back to /r/mensrights

>>22>>23
lol, she thinks she won't lose her job for being a sexist pig with ugly penis.

One of British soccer's leading television commentators was fired Tuesday, a day after being taken off the air and temporarily suspended for making sexist remarks about a female match official.

Andy Gray, the face of Sky Sports' soccer coverage for the past two decades, was dismissed by the broadcaster after "new evidence of unacceptable and offensive behavior" that took place off-air last month.

The former Scotland striker and broadcast colleague Richard Keys had been reprimanded and removed from duty Monday for making derogatory comments about lineswoman Sian Massey, former referee Wendy Toms and West Ham executive Karren Brady.

In an off-air exchange with Andy Gray, Keys commented that "Somebody better get down there and explain offside to her." After Gray suggested that "Women don't know the offside rule", Keys remarked "Course they don't. I can guarantee you there will be a big one today.

>>25
Shut up JEWS boy.
Nobody in here gives a shit about your autism.

Yeah, feminist trolls in /prog/ !

lol "she"

hate when programming-blog-fags use it, like the typical neckbeard  computer programmer reading it is female

>>28
your neckbeard is sexist.

Xarn is of the kind who would alternate ``she'' and ``he'' so don't laugh at it, fuckers.

I always use she, it's funny because the reader has a vagina

hax my neckvagina