[xquery-talk] Izzit Bcos I is functional?

Thu Jun 18 15:15:07 PDT 2015

On Thu, Jun 18, 2015 at 12:23 PM, daniela florescu <dflorescu at me.com> wrote:

> Ihe,
>
> the discussion starts to take too many threads for my brain the be able to
> wrap around it.
> But I’ll answer some of the questions that  came up in your emails, one by
> one.
>
> 1. Scala WILL be used. Spark is implemented in Scala. So there will be
> millions of Scala instances
> running.
>
>
Scala is a language for which you do need a CS degree. When I say that I
don't literally mean you need a degree but you need knowledge that you are
less likely to have encountered if you have not done a CS degree.

In general functional programming is something you need to learn at college
or at least from an academic, not from other developers and especially not
from non-academics who have written books because they don't know what to
teach or how to teach it.

So where are these Scala programmers going to come from. Well they need to
come from the universities but I don't think universities will produce them
because universities don't like failing students. Back in the 90's FP was
on the 1st year curriculum at UCL. I asked my tutor why it got removed, he
said too many students were failing it and  passing the 1st year
programming course was mandatory to getting to year 2, so they changed it.
Happily they have reverted and now teach Haskell in year one. Programming
Scala will be a bloody hard course I don't think enough students
could/would get through it (stayed tune for another post on this).

> If ... Typesafe (the company who does the Scala compiler) will make money
> out of this: I would say no.
> I followed their business plan…. and honestly, I am not impressed. I think
> they have very good engineers but no clue
> how to turn this into money. And that’s unfortunate because they COULD
> make money out of it,
>
> 2. About what’s more important for the success of a programming language:
> #installations, # developers or #millions$$.
>
> Well, it’s subjective.
>
> Honestly, all I care is that (a) there is an important problem that people
> need to solve and (b) that programming language is the
> best tool to solve that problem.
>
> According to this, yes, XSLT and XQuery do the job.
>
> And, also, I don’t think either XSLT or XQuery or JSOniq are the kind of
> languages that increase the number of developers to millions.
>

Agree.

>
> 3. The idea that we need “millions of developers” gets me crazy. No, we
> don’t.  We just need smarter people with better tools.
> (A good tool like Oxygen has more effect on the success of XQuery then
> anything we cold have done to design it better….)
>
>
Agree.

> E.g. today there was an article that made the tour of the Internet: "US
> needs 5 million data scientists by 2017.”
>
> Now contrast that with the little experiment that I was running recently:
> here is the simplest statistical problem you can imagine.
>
> You have 3 boxes and 6 balls: 3 white and 3 back. You put the balls in the
> boxes, 2 in each. I take one ball out of one box and it is black.
> What is the probability that the secod ball in the same box is also black ?
>
> I asked this question to CTOs of data science companies, Phd in math, Phd
> In CS, the head of machine learning at Google, hordes of “data scientists”.
>
> Out of tens of people I asked, I got only TWO correct answers: both where
> Phd in physics. (not the CTOs, not the head of machine learning..)
>

Pick a Box A.

Sampling is without replaceent.
Chance of 1st ball being black = 3/6 = 1/2
Chance of 2nd ball being black = 2/5.

Chance of both being blank = 1//2 * 2/5 = 1/5.

BUT there are 2 ways for Box A to have 2 black balls

a). Box B could have zero black and Box C 1 black  or
b) Box C could have zero black and Box B 1 white.
So multiply by 2

Se P (Box A has 2 black balls) = 1/5 * 2 = 2/5.

BUT there is only a one in 3  chance of selecting Box A

Final answer = 2/5 * 1/3 = 2/15

I settle for partial credit.

> Now, tell me : if US hires 5 million (!!) data scientists in the next 2
> years, what’s the probability that ANY good data science, and some reliable
> results come out of that! ?
>
> You know the story: garbage in, garbage out ?
>
> So, no I don’t believe that hiring hordes of uneducated developers helps.
> (maybe it’s my ex-communist snobbish view concerning the education, or lack
> thereof..)
>

> Just hire less developers, but smarter ones, and give them very good tools
> to do their job, that’s all.
>

Agreed

On Thu, Jun 18, 2015 at 12:23 PM, daniela florescu <dflorescu at me.com> wrote:

> Ihe,
>
> the discussion starts to take too many threads for my brain the be able to
> wrap around it.
> But I’ll answer some of the questions that  came up in your emails, one by
> one.
>
> 1. Scala WILL be used. Spark is implemented in Scala. So there will be
> millions of Scala instances
> running.
>
> If ... Typesafe (the company who does the Scala compiler) will make money
> out of this: I would say no.
> I followed their business plan…. and honestly, I am not impressed. I think
> they have very good engineers but no clue
> how to turn this into money. And that’s unfortunate because they COULD
> make money out of it,
>
> 2. About what’s more important for the success of a programming language:
> #installations, # developers or #millions$$.
>
> Well, it’s subjective.
>
> Honestly, all I care is that (a) there is an important problem that people
> need to solve and (b) that programming language is the
> best tool to solve that problem.
>
> According to this, yes, XSLT and XQuery do the job.
>
> And, also, I don’t think either XSLT or XQuery or JSOniq are the kind of
> languages that increase the number of developers to millions.
>
> 3. The idea that we need “millions of developers” gets me crazy. No, we
> don’t.  We just need smarter people with better tools.
> (A good tool like Oxygen has more effect on the success of XQuery then
> anything we cold have done to design it better….)
>
> E.g. today there was an article that made the tour of the Internet: "US
> needs 5 million data scientists by 2017.”
>
> Now contrast that with the little experiment that I was running recently:
> here is the simplest statistical problem you can imagine.
>
> You have 3 boxes and 6 balls: 3 white and 3 back. You put the balls in the
> boxes, 2 in each. I take one ball out of one box and it is black.
> What is the probability that the secod ball in the same box is also black ?
>
> I asked this question to CTOs of data science companies, Phd in math, Phd
> In CS, the head of machine learning at Google, hordes of “data scientists”.
>
> Out of tens of people I asked, I got only TWO correct answers: both where
> Phd in physics. (not the CTOs, not the head of machine learning..)
>
> Now, tell me : if US hires 5 million (!!) data scientists in the next 2
> years, what’s the probability that ANY good data science, and some reliable
> results come out of that! ?
>
> You know the story: garbage in, garbage out ?
>
> So, no I don’t believe that hiring hordes of uneducated developers helps.
> (maybe it’s my ex-communist snobbish view concerning the education, or lack
> thereof..)
>
> Just hire less developers, but smarter ones, and give them very good tools
> to do their job, that’s all.
>
> 4. However, even I don’t believe that we need millions of developers
> writing XQuery, the question still remains:
>
> Did we do the best we can to market XQuery ? Of course not. Books are
> scarce, the little material is badly written, there are no classes, etc.
>
> The W3C XQuery web site had “to be added..” for 8 YEARS !!!!! (don’t even
> get me started …)
>
> The community is split, and instead of getting together to try to make a
> common effort to market their common interest, they each one fights for
> showing off THEIR muscles, etc.
>
> The XQuery community spent no efforts into marketing, and unfortunately a
> programming language, like any product, needs MARKETING.
>
> The SQL community spent TONS of efforts for marketing it… long ago….and
> now you can see the results.
>
> Scala community spends TONS of efforts for marketing. Etc. Etc.
>
> Well. I am not sure how to solve that, other then to open my (big) mouth
> every time I get a chance. (and yes, usually the results is… "Dana is
> grumpy" :-)
>
> A more community-based effort would be needed to  better explain to the
> world what XQuery is and what they can do with it.
>
> And yes the fact that MarkLogic pretends that they’ve never heard of
> XQuery in their life and sells this as “their” technology
>  to their customers  doesn’t help XQuery either. (and honestly, I think
> it’s dumb business mistake for them too….but  who am I to judge....).
>
>
>
> ===========
>
>
>
> A separate email about the market of XQuery.
>
> Best
> Dana
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Jun 18, 2015, at 12:07 AM, Ihe Onwuka <ihe.onwuka at gmail.com> wrote:
>
>
> On Wed, Jun 17, 2015 at 2:01 PM, daniela florescu <dflorescu at me.com>
> wrote:
>
>> Depends what you call “mainstream” (or successful) for a programming
>> language…...
>>
>> Is it calculated in (1) #developers ?
>>
>> Or is it calculated in (2)  #instances running it ?
>>
>> Or is it calculated in (3) #millions of dollars revenue from it ?
>>
>> ==========
>>
>> A. If (2), the Scala will be a HUGE success — because of Spark.
>>
>> IBM just offered to put more then 3000 (!! and that’s a LARGE number) of
>> developers on Spark.
>>
>>
> Yes on Spark. Not Scala. Spark.
>
> Cloudera just seemed to have dumped Hadoop to become a Spark reseller.
>>
>> So…. in terms on number of instances running it, Scala WILL be a huge
>> success.
>>
>>
> Spark supports Java, Python and Scala there is an open source SparkR.
>
> Now let me remind you of your opening gambit in the discussion.
>
> ************************************************************
> "Even though, because it’s functional, it will be restricted to be used
> only by people with CS degrees, and not by
> random Joes and Janes who write web sites. The way it is designed it is
> intended to make a population of educated programmers
> ETREMELY efficient, and NOT to increase the  total number of developers to
> hundreds of millions.
>
> When being reproached this fact in the past, my answer was always the
> same: building a database application should not be for the uneducated.
> It’s like building a 30 story building, you don’t do that without a
> architect ad a structural engineer.
> E.g. if you want to eradicate a grave neurological disease, you don’t
> lower the bar to allow anyone from the street to perform a neurosurgery,
> you just make the existing neurosurgents more productive."
> *************************************************************
>
> Now you tell me what you think is going to happen.
>
> (how is Tyrpesafe, the company that does the Scala compiler making money …
>> that’s another story…)
>>
>> B. If it is (3) — millions of $$$, I can probably give you the most
>> “cost-effective” programming language in the world.
>>
>> https://en.wikipedia.org/wiki/K_(programming_language)
>>
>> It’s used for high end financial systems, and a copy of this compiler
>> costs about $1M.
>>
>> Not bad ….
>>
>> Number of developers writing code in this language … probably in the
>> 100s….
>>
>> C. If it in terms of (1) number of developers …. are we really sure it
>> matters ?
>>
>>
> Yes.
>
>
>
>> Who makes money today out of Javascript ? (arguably the most popular
>> programming language …)
>>
>> Nobody.
>>
>>
> Loads of people make a living out of it.
>
>
>> ======================
>>
>> So, which one do you care about ?
>>
>> I care about (2) and (3).
>>
>>
> If the answer were 2 XML would be deemed to be a raging success. But of
> XQuery's woes you said "So…. I think it is simply a question of …. there
> is no market for XML ……(aka no enough MONEY in the market)."
>
> Similarly having loads of instances running Scala has to translate in some
> meaningful way (beyond making money for Databricks and co). Show me the
> money.
>
>
>
>> On Jun 17, 2015, at 10:47 AM, Ihe Onwuka <ihe.onwuka at gmail.com> wrote:
>>
>> I wish I could agree with you but I think it is different this time.
>>
>> Couple of days ago I saw an update on the Scala group, somebody saying
>> that the upsurge of interest in Spark could be the killer app that
>> catapults Scala into the mainstream. Much as I would like to see it happen
>> for a functional programming language, everybody except Scala dev's knows
>> that the language is just too damn complicated to ever go mainstream.
>>
>> Even if this criticism could not be levelled at Scala, suspend disbelief
>> for a minute and accept that Spark is indeed this killer app, alas it is
>> not going to catapult Scala anywhere because the people employed in that
>> domain will demand that it is delivered in Python and/or R and the people
>> that hire them will acquiesce and say verily so.
>>
>> In the 1990's the bell tolled for the Cobol mainframe programmer. It's
>> not like that now. On the JVM, the message is not adapt to Scala/Clojure or
>> die it's don't worry mate stick to what you know and Java 8 will bail us
>> out.
>>
>> The IT industry has presided over the widepsread and rudimentary
>> amateurisation of  software development. So when the right solution comes
>> along it encounters a rearguard  resistance from people who depend on the
>> technological status quo for their jobs and who roll out their stock in
>> trade objections (performance usually high up on the list). It's not like
>> 1990 when mainframe programmers were saying I need to learn Unix/C and an
>> Rdb and/or 5 years later I need to learn Java and what they thought to be
>> OOP. Hence 20 years later we are still talking Java, Javascript and SQL and
>> 5 years on they will be looking at Java 10 and still writing Fortran with
>> classes. The industry goes along with it because they can continue to
>> source bodies cheaply.
>>
>> I absolutely agree that what you said should be the way it is goes but I
>> don't see how it is going to happen with the vested agenda's at play.
>>
>>
>>
>>
>>
>>
>> On Wed, Jun 17, 2015 at 1:10 PM, daniela florescu <dflorescu at me.com>
>> wrote:
>>
>>>
>>>>
>>> Well I have no particular beef with the format itself other than the
>>> lack of tools. Now that we have JSONiq I am less bothered about that
>>> (assuming one has the opportunity to use it).
>>>
>>>
>>> Well, JSONiq is only implemented by Zorba (and another implementation in
>>> IBM middle tier).
>>>
>>> And Zorba is a dead piece of code.
>>>
>>> So, having “JSONiq” as a specification…...doesn’t mean much, isn’t it ?
>>> Unless is adopted by other XQuery processors.
>>> (which I cross fingers they will do…)
>>>
>>>
>>>
>>> I agree with your ideals (1 and 2 above) too but it should be evident
>>> from the sociology of the JSON community that these things are not going to
>>> happen.
>>>
>>>
>>> Well… nope. Not clear at all.
>>>
>>> I started working on query languages for XML in 1996.
>>>
>>> Same as now, the whole industry was for using SQL for querying XML —
>>> including ME, I had
>>> a system running, a bunch of PhD students working on that, etc.
>>>
>>> The decision of the W3C NOT to use SQL for that purpose was taken in
>>> 2001.
>>>
>>> 5 years later. You know how many query languages have been proposed
>>> during those 5 years ?
>>> Tons: UnQL, XML-QL, etc, etc.
>>>
>>> Those things need TIME.
>>>
>>> People need to try SQL first, before they realize it’s a dead end.
>>>
>>> MarkLogic needs to try Javascript on the server side, before realizes
>>> that’s a dead end.
>>>
>>> The industry moves MUCH, MUCH slower that one can expect.
>>>
>>>
>>> You have people putting stuff in JSON databases without thinking how are
>>> we going to get it out and coming up with half-assed solutions for doing
>>> so. This is not progress and this is not good.
>>>
>>>
>>> Again, patience is golden :-)
>>>
>>> There will be tons of those half baked solutions (MongoDB’s JSON query
>>> language, CouchDB’s..), before people realize that this
>>>  is not going anywhere….some of those databases will be acquired, never
>>> to be seen again, them or their query languages….etc.
>>>
>>> ===============
>>>
>>> Honestly, I give it 5 years for the JSON query languages to stabilize.
>>> 2020 would be my estimate. Even later if there is a database bubble crash
>>> in the
>>> meantime.
>>>
>>> Best
>>> Dana
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://x-query.com/pipermail/talk/attachments/20150618/4c9e60b5/attachment.html>