If you run more than one test, you are bound to eventually get a false positive significant result.
I don't know where I'm going with this. I'm using AI a lot myself, always supervised. This hits different.
I have seen plenty of overoptimistic results due to improper building of training, validation and test sets, or using bad metrics to evaluate trained models.
It is not clear to me that this project is going to help to overcome those challenges and I am a bit concerned that if this project or similar ones become popular then these problems may become more prevalent.
Another concern is that usually the "customer" asking the question wants a specific result (something significant, some correlation...). If through an LLM connected to this tool my customer finds something that it is wrong but aligned with what he/she wants, as a data scientist/statistician I will have the challenge to make the customer understand that the LLM gave a wrong answer, more work for me.
Maybe with some well-behaved datasets and with proper context this project becomes very useful, we will see :-)
> RMCP has been tested with real-world scenarios achieving 100% success rate: