ilo Token is a rule-based dictionary-based toki pona to English translator that translates into multiple English outputs showing many possible grammatical and semantic interpretation of the text. It is work in progress. Please read the limitation when using it.
9 Likes
It is WORKING!
Image description
Welcome to the ilo Token REPL. Press ctrl+d or ctrl+c to exit.
mi pona
- I am good
- I was good
- I will be good
- I am positive quality
- I was positive quality
- I will be positive quality
- I am positive qualities
- I was positive qualities
- I will be positive qualities
- We are good
- We was good
- We will be good
- We are positive quality
- We was positive quality
- We will be positive quality
- We are positive qualities
- We was positive qualities
- We will be positive qualities
- Good me
- Good us
- Me of positive quality
- Us of positive quality
- Me of positive qualities
- Us of positive qualities
I need to work on other things for now, e.g. preposition translation. Then it'll be released. I'M SO EXCITED!
Now off to lunch
5 Likes
oh, there's "were" instead of "was", there are also other bugs of course, will fix those
1 Like
Signs of burnout has been showing, so I'm going to release this early. No prepositions 
sorry. Expect another hiatus.
Please don't underestimate how huge this project is, this took me 8510 lines of codes.
0.4.0 has been released, please check the changelog for more information.
Testing it out, giving feedback, and reporting bugs would greatly help with this project. This is mainly untested lmao.
1 Like
this is really neat and quite an impressive project. i was thinking about doing a similar thing but only in reverse direction. (actually maybe an input language specifically designed for being translated into other languages by a rule-based translator would be a good idea.) my logic is that rule-based english sentences would be easier to translate into toki pona so that there doesn't have to be more than 1 output sentence. (i was thinking that maybe there could be an option for tweaking the output verbosity... like tablecloth into "len" or "len supa" or "len pi supa moku"...) i thought that what you did would be harder because of the large amount of possible outputs.
i guess you're right that creating an english parser is probably very hard... but maybe if there were suficcient limitations. or maybe with some specific subset of english (like basic english or something).
Don't take my word for it. I was hugely unfamiliar with NLP when I wrote that FAQ and I only came from experience with parsing computer-readable language. there are NLP tools that makes it easy to parse English. However, the translation side of things, I'm unsure. sona pona says you'll need semantic decomposition algorithm. Or yeah, you could limit it to basic English.
I just updated the FAQ
1 Like
it's a very cool tool! i was always thinking of doing this in the past and i'm very glad it's actually being made!
2 Likes
I've also had concepts of this in the past, and I'm really excited someone's making it. I've been following this project for some time, and can't wait until it can do even more!
1 Like
Thank you all so much for the kind words
1 Like
Idea for "la" just use adverbs:
"lon la X" → "truthfully, X"
"kin la X" → "additionally, X"
I can't believe I just figured this out
0.4.1 has now been released, no new features has been added. only bug-fixes and performance improvement. Umm not really a performance improvement, memory usage has been sacrificed. See the changelog
0.5.0 has now been release, ilo Token can now translate "o" imperative sentences
1 Like
it cannot do preverbs yet
1 Like
sorry i completely forgot to put text on that message. i just wanted to say i find it funny how vaguely accurate that is
2 Likes