A polarized nation

Eugen G Tarnow  October 1 2018 09:53:10 AM

How can we measure the polarization of the US? One way is to assume that it is reflected in Congress. There is a measure of the polarization in Congress called the "DW-nominate score" difference between the two parties.  It is indicated below.

It seems that in the absence of great misery or external enemies, our nation polarizes - in other words it follows typical group dynamics.



Numbers read off from here and for more information about the DW-nominate score see here.

 
Comments Disabled

Drug barely better than placebo is supposed to save TEVA

Eugen G Tarnow  September 29 2018 10:14:13 AM

According to the Jerusalem Post:

"Only two weeks ago, Teva received approval for its newly developed migraine drug known as Ajovy, raising hopes of improved financial fortunes for the heavily indebted Israeli company. Ajovy could bring in as much as $500 million in sales annually, according to Deutsche Bank estimates. --- The drug's monthly list price is an identical $575, or $6,900 annually."

If we take a closer look at the efficacy of this drug we see that the number of migraine days in a month goes down by 3.5 days. Placebo takes care of 2.5 days.

This is the state of Big Pharma today: make fortunes off of stuff that is basically snake oil. And the US FDA is an active participant.

I could not find the number to treat. Presumably it is pretty high.

But the graph also suggests that somebody ought to set up a website like this one and charge 2/3 of the price, deliver a saline injection, and the patients would be getting the same bang for the buck.
Comments Disabled

When Google’s Algorithm & Wikipedia Editors Fail

Eugen G Tarnow  September 17 2018 07:51:11 AM

In order to find out just how much money is wasted on ineffective drugs (see this), I was looking for sales numbers of the particular drugs. I was then surprised to find out just how much the Google algorithm favors the site drugs.com , which I have previously shown to have suspicious drug ratings.

All that work and Google can't get it right.

True, neither can Wikipedia. Together they work to boost drugs.com ..
Comments Disabled

Open Letter to the US Patents & Trademarks Office (USPTO)

Eugen G Tarnow  September 13 2018 10:53:52 AM

To Whom It May Concern:

It is almost impossible to be a small inventor and deal with the USPTO.

The Electronic Filing System (EFS) software you provide is so complex that it is unusable. The software allows you to submit incomplete applications (which you will then penalize inventors for). It also does not provide a record of what you actually submitted. If you want to pay fees, you cannot click on the fees tab but you have to pretend to be uploading documents.

And then once in a while you get a gem. The Electronic Filing System tells you that you will receive something in the mail:


What did I get? Another notification that, as far as I could see, could have been delivered "in electronic form".

The correspondence you send out is not human readable. It is all about the patent numbers instead of the invention titles and about bureaucratic minutiae that you don't bother to explain.

In order to deal with you I have to know three different phone numbers because each department does not know anything about what the other departments do. For example the Application Assistance Unit knows nothing about EFS. And then when I complain to your Commissioner's office you give me two new places to call on (one of which is the Pro Bono program. So in addition to tax payer funding, you give out others people's time. In any case this program has one participant in New Jersey and they only handle starving artists. Kid you not. So that was one useless lead.)

In other words, even if you work at the USPTO full time you have no idea how to go through what a small inventor goes through.

And you always provide the same "Our customer center is experiencing longer wait time today". Which days are you experiencing shorter wait times?

Your fees are really high. $750 to apply for a patent - when only 1% of patents make any money, thus requiring a minimum of $75,000. You better have an investor just to deal with the USPTO. Never mind marketing or actually making the products.

You assess huge penalties for late responses to trivial issues.

I use lawyers to help me prepare documents and then file them myself. I then get a letter from the USPTO telling me I should use a lawyer. In other words, the message only depends upon who files it, not what the content is.
Comments Disabled

In France, Young Scientists Suffer Just The Same

Eugen G Tarnow  August 29 2018 04:45:39 PM

I am vacationing on the French Riviera and had an occasion to talk to a PhD student in a physical science. I asked him if he knew about the job prospects after graduation. He did; there was a 1% chance that he would get a job in his field.

He remarked that he only found that out in his last year of getting his masters degree.

So in France, young scientists are cannon fodder just like in the US.


Comments Disabled

The great sadness that goes with the American Psychological Associations Style Manual

Eugen G Tarnow  August 29 2018 04:22:23 PM

We already know that social science, and psychology in particular, is suffering from a problem of replicating reported findings - in other words - what goes into the psychology research journals cannot be trusted.

But I am going to go one step further: I believe social science in the US is at such a low point that one would be hard pressed to see a difference from Communist Russia. Only certain people get published (just like in Communist Russia only writers who were in the Writers Union were published) and quality and success can be perpendicular.

The other day I had some time to read a prestigious social science journal and I fell off my chair several times when realizing just how bad the articles were. Claims were made about short men that when scrutinized showed a tiny effect size, a claim was made that the happiness from receiving presents is determined by social class but this so called determination also originated in a very small effect size and then gentlemen from Yale compared racism to opinions of chemical mixtures including urine and cyanide. I kid you not - here is the link.

The closest analogy with Communist Russia that can be easily documented is that the "style" of the writing is imposed from above - it is decided by bureaucrats rather than by the authors and editors themselves..

Thus the American Psychological Association, alone among all scientific societies I know of, publishes a style manual.

Interestingly, a new finding about Grievance Studies ( https://youtu.be/kVk9a5Jcd1k ) shows that if complete crap is published in APA format and using the terms of art, articles are likely published with complete disregard to the actual content - even Mein Kampf drivel is accepted.

I am not sure how many readers actually read psychology research journals but the APA Style manual (with its own website apastyle.org!) is trying its best to prevent them from doing so. The APA style manual has made research articles several times longer than they used to be (the APA style manual functions in an environment with no consequence for excessive erudition). One can read an introduction over a lunch only to find out over coffee that the results did not justify any reading at all.

Just one example that would baffle anyone outside the field, the guidelines for the Methods section - how many words do you think compliance with this will require:

"Participant characteristics Eligibility and exclusion criteria, including any restrictions based on demographic
characteristics
Major demographic characteristics as well as important topic-specific characteristics (e.g.,
achievement level in studies of educational interventions), or in the case of animal
research, genus and species
Sampling procedures Procedures for selecting participants, including:
The sampling method if a systematic sampling plan was implemented
Percentage of sample approached that participated
Self-selection (either by individuals or units, such as schools or clinics)
Settings and locations where data were collected
Agreements and payments made to participants
Institutional review board agreements, ethical standards met, safety monitoring
Sample size, power, and
precision
Intended sample size
Actual sample size, if different from intended sample size
How sample size was determined:
Power analysis, or methods used to determine precision of parameter estimates
Explanation of any interim analyses and stopping rules
Measures and covariates Definitions of all primary and secondary measures and covariates:
Include measures collected but not included in this report
Methods used to collect data
Methods used to enhance the quality of measurements:
Training and reliability of data collectors
Use of multiple observations
Information on validated or ad hoc instruments created for individual studies, for example,
psychometric and biometric properties
Research design Whether conditions were manipulated or naturally observed
Type of research design; provided in Table 3 are modules for:
Randomized experiments (Module A1)
Quasi-experiments (Module A2)
Other designs would have different reporting needs associated with them "

The APA Style Manual and its imposition on journals (and thereby authors) in order for the journals to be included in PsychInfo is nothing but an experiment in "will anyone ever say no"? Stanley Milgram asked this question of people giving each other electrical shocks 56 years ago and found that few people would say no.

The Milgram obedience experimental result has never changed.

But even Milgram would have been sad that among psychology researchers NOT ONE has ever told the APA Style Manual and its enforcers to fuck off.
Comments Disabled

Economists suddenly find out that the large companies are hogging all the productivity gains

Eugen G Tarnow  July 15 2018 07:08:23 PM

Economists seem to be divorced from the reality nowadays.  Recently they found out that the large companies are hogging all the productivity gains (see https://www.wsj.com/articles/the-problem-with-innovation-the-biggest-companies-are-hogging-all-the-gains-1531680310?mod=hp_lead_pos5).

Really? Now they find that out? Ever since the monopolies realized they did not have to have a support phone number they started foisting the work onto the users.  If there is no competition, the users cannot complain. So Microsoft never gives public support and we all have to spend many hours a year trying to figure out what they want us to do.

No wonder Microsoft is more productive.

Google has taken over the internet to such an extent that if you look for reviews of website IT IS ALL ABOUT MAKING THE WEBSITES NICE FOR GOOGLE, NOT FOR THE HUMAN EYEBALLS!

No wonder Google is more productive - every small business is working for them for free.

I have previously written about Amazon trying to suppress negative reviews and one has to figure out what their algorithm is if one wants to post a negative review.

The latest example I noticed is that if one wants to put up a youtube video one cannot choose the thumbnail! Unless one is willing to have Google ads displaying for at least a year and create a popular enough movie that 1000 people watch it.

No wonder YouTube is more productive - they expect lots of free labor from us. Here is one example: https://www.youtube.com/watch?v=KuPn4_i7c64

Economists - get your heads out of your butts.

And politicians - pledge you will no longer take money from monopolies!
Comments Disabled

Harvard tells Asian-Amerians - you do not have a positive personality

Eugen G Tarnow  June 15 2018 01:01:06 PM

Harvard is being sued for their Asian-American quota system.  

According to a Wall Street Journal article they currently implement their quota by ranking Asian-Americans low on "positive personality".

I took a graduate course in psychology from a great clinical psychologist at Harvard in the end of the 1980s, George Goethals. He acknowledged the existence of the quotas.

But the rest of the University has been unable to fess up.

This is particularly poignant in light of Harvard's motto:

"Veritas".
Comments Disabled

This is what your degree gets you

Eugen G Tarnow  June 2 2018 08:02:18 AM

Comments Disabled

Bezos and his minions

Eugen G Tarnow  April 23 2018 09:51:11 AM

Our work was covered by WSJ Daily Shot, see https://blogs.wsj.com/dailyshot/2018/04/24/the-daily-shot-job-hopping-is-becoming-increasingly-lucrative/
Comments Disabled

EZ

Eugen G Tarnow  April 11 2018 06:34:42 PM

Comments Disabled

How good are the top selling drugs?

Eugen G Tarnow  April 5 2018 10:31:22 AM

Comments Disabled

WSJ’s Daily Shot covered our research - School District Income Seems to Mean Little to the Poor

Eugen G Tarnow  March 9 2018 07:42:05 AM

See http://mitphd.com/check%20big%20data%20blog.nsf/dx/school-district-income-seems-to-mean-little-to-the-poor

and go to the bottom at:

https://blogs.wsj.com/dailyshot/2018/03/08/the-daily-shot-federal-government-now-owns-over-30-of-all-u-s-consumer-debt/#FoodforThought
Comments Disabled

Deaths from gun violence & oldfashioned IV bags

Eugen G Tarnow  March 1 2018 08:14:11 AM

Comments Disabled

WSJ’s Daily Shot covered our research - Facebook, Twitter, YouTube & Instagram in Mueller Indictment

Eugen G Tarnow  February 17 2018 01:21:06 PM

https://blogs.wsj.com/dailyshot/2018/02/19/the-daily-shot-weak-dollar-sends-u-s-import-prices-soaring/
Comments Disabled

WSJ’s Daily Shot covered our research - make sure to go to college if you expect to work at Amazon

Eugen G Tarnow  February 12 2018 08:44:54 AM

https://blogs.wsj.com/dailyshot/2018/02/12/the-daily-shot-fiscal-stimulus-rivaled-only-by-the-korean-and-vietnam-wars/
Comments Disabled

Don’t finish high school if you want to attend City University of New York

Eugen G Tarnow  February 12 2018 08:43:57 AM

Comments Disabled

Google - the new drug dealer on the internet?

Eugen G Tarnow  January 30 2018 02:27:35 PM

Here Google is selling opioids::


Comments Disabled

Reverse Machine Learning

Eugen G Tarnow  January 29 2018 12:10:21 PM

I have coined the term "reverse machine learning" earlier in this blog.  The latest example comes from an incorrect meter reading by PSE&G, the electrical utility for our office.  

The person who came out misread the meter and we go a huge bill, 70 times larger than it should have been.

I figured that their computer system just does not catch the problem (their programmers are just stupid?)  and their human labor is not properly supervised.

I decided to test it.  I tried to enter a correct meter reading, which would be considerably lower than the previous human meter reading.  That did not work.  I got an error message:



In other words, the PSEG computer system presumably (with that error message - who knows?) insists on a higher meter reading than the last reading if I read the meter but on purpose lets the outlier meter reading of their employee pass.

Is this fraud on the part of PSEG? Who knows.  But it certainly is reverse machine learning - the computer knows something we do not and refuses to tell us what.



Comments Disabled

WSJ’s Daily Shot covered our research - the riches of Putin, Gates & Trump

Eugen G Tarnow  January 9 2018 08:46:59 PM

See https://blogs.wsj.com/dailyshot/2018/01/12/the-daily-shot-u-s-household-leverage-is-much-higher-than-previously-reported/#FoodforThought
Comments Disabled

WSJ’s Daily Shot covered our research - a society with no children?

Eugen G Tarnow  December 15 2017 08:41:37 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

The Wall Street Journal's Daily Shot covered our research.  See https://blogs.wsj.com/dailyshot/2017/12/15/the-daily-shot-median-household-net-worth-has-languished-since-the-recession/#FoodforThought and scroll to "Avalon".

The Daily Shot had shown that the workforce participation of women in Norway is higher than in the US. The Scandinavian countries always have an air of superiority and I decided to check it out.  It turns out that for each increase in workforce participation in Norway over US, there was a similar decrease in the number of children.  In other words, Norwegian women are not any more hard working than US women, there is simply a tradeoff between having children and going to work.

This trade-off is a world-wide phenomenon. Below are the graphs showing that as the workforce participation over the last 25 years increases, the number of children decreases.  The left graph shows the number of kids below 5 years of age, the right one the number of kids being born. In either case, an extrapolation shows that we will have no kids when the women work force participation is 59-60%.and the same extrapolation suggests it will happen 50 years from now.  If about two kids is necessary for replacement (I never quite understood this factoid) then our global population will start to not replace itself in about 12 years.

If we consider all the possibilities of the ending of the human race, nuclear war, supervolcanos, asteroids, global warming, etc., our decision to not have children is definitely up there.

Perhaps men need to start to work less and take over the raising of kids?

Of course, extrapolations do not take into account system changes, but it is food for thought.



As always, I reserve the right to be wrong.
Comments Disabled

12-50 flu shots to prevent one flu

Eugen G Tarnow  December 9 2017 08:41:58 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

Every year we are told by our health professionals and the media to get the flu vaccine.

What might be the statistics of the flu shot?

According to my average of CDC numbers the average efficacy is 42% (2005-2016, not including 2009).  The flu shot gives you a 42% lower chance of getting the flu than if you did not get the shot.  

The chance of getting the flu is not that high, it is 5-20% according to WebMD.  

So if you do pay for the flu shot you lower the probability of getting the flu from 5-20% to 3-12%.

In the best case scenario you wait twelve years to prevent one flu, in the worst case scenario 50 years.  12-50 shots to prevent one flu.

I used the term "my average" - as if averages would be different depending upon the person doing the calculation.  It turns out they are.

CDC states, contrary to their own numbers, that "recent studies show vaccine reduces the risk of flu illness by about 50%  to 60%".  50-60% is larger than 42% and there is actually no way to get the average into the 50-60% range - no matter how many or few years you use (as long as you go chronologically from the 2016 number).  WebMD claimes that CDC states a 70-90% efficacy.

Comments Disabled

A highly improbable set of Amazon reviews

Eugen G Tarnow  November 28 2017 09:13:58 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

I just came upon this gadget listed on Amazon.  It has 330 reviews, all of them 5 stars.

What is the probability of finding 330 people all agreeing on 5 stars?


Comments Disabled

Why US small business cannot compete with China - hint - it is the postal service...

Eugen G Tarnow  October 23 2017 12:00:33 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

Why is that that you can buy Chinese stuff so cheaply on eBay and it arrives with no additional shipping charge?  Because it is a lot cheaper to mail something from China to the US than vice versa:


Source: skubana.com

The delivery fee that US charges China is $2 per pound by international agreement.  This is similar as for other countries.  However, this system was based on the mutual package delivery being roughly the same both ways.

But, it is not.

10/17/18: THANK YOU PRESIDENT TRUMP: https://www.wsj.com/articles/trump-administration-moves-to-end-deep-discounts-for-packages-from-china-1539791629?mod=hp_lead_pos6
Comments Disabled

The Relationship of College Degree to Current Job

Eugen G Tarnow  October 16 2017 10:59:56 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

Is college a great way to prepare for a job or is it a yoke for the youth to bear to allow them membership in adult society?

300 alumni of a large private university, members of an honor society, having completed their BS, MA, or PhD between 1983-2016 were surveyed as to the relationship of their college degree to their current job.

81% of respondents deemed the college degree to be necessary to obtain their current job.



On average 71% of the college degree was judged as relevant to the current job (see figure below). The mode was at 100% and the distribution had a very large standard deviation (34%) and range (from 0% to 100%).  An ANOVA showed a very large division between those who did not deem the college degree to be necessary for their current job (25% of college degree useful) and those who did (78% of college degree useful). For those whose jobs required the degree the range was large (0-100%) but the standard deviation somewhat smaller (28%).



The single most useful course in the major was typically a second year course (average year level was 2.3).

A 19% response rate was obtained.
Comments Disabled

Will the company cease to exist in 2024?

Eugen G Tarnow  September 24 2017 06:16:59 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com


Linear extrapolations may not be the most accurate ways to predict the future but here is an example of a company for which TWO linear extrapolations yield the same result: that the company should cease to exist in 2024:



There is a discrepancy - however - the market capitalization is much higher than the sum of future profits.




Comments Disabled

Chinese Mobike in Washington, D.C.: An Excellent Opportunity for Gatheting Information on Policy Makers

Eugen G Tarnow  September 21 2017 10:29:27 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com


The Chinese company Mobike is renting out bicycles in the capital.  

The starting and end points of all bike routes are marked with the identity of the cyclist.   And the trips are smartphone activated...

I am not familiar with Washington's goings on but I would presume this would yield a lot of intelligence on who is working with whom on various policy issues.

Imagine what we could learn about North Korea if we rented out bicycles there?

Comments Disabled

Household incomes: the tail is getting longer

Eugen G Tarnow  September 19 2017 12:07:20 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

It is frequently stated that income inequality is increasing.  Measuring this is more difficult.  Here I use one measure - the tail of the high-end household income distribution (inflation adjusted to 2017 dollars).  

This tail is well defined for census data between $60,000-$200,000 and approximately exponential:



I can express the exponential in terms of the income difference at which the number of households decrease by half:



What we see is that the high end tail is getting somewhat wider.

(There are two points for 2013 because the US Census Bureau changed the way the census was done and there are two different results. )
Comments Disabled

How come the credit rating agencies did not flag the Wells Fargo account fraud?

Eugen G Tarnow  September 16 2017 08:49:07 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

Equifax is in the news for a colossal security breach. It seems the customers who paid for credit security were the hardest hit.

But this may not be the biggest scandal on the horizon involving the credit rating agencies, Equifax, Experian, TransUnion, etc.

The biggest news story might be that the credit rating agencies must have known about the Wells Fargo account fraud.  

Imagine that suddenly one institution is creating lots of new accounts for their customers, all of which should show up in the credit rating data. Wouldn't you think credit agencies would notice it?

There are two possibilities: (1) the credit rating agencies did know about it and chose to keep quiet or (2) the credit rating agencies are somewhat incompetent.

Let's see which way it is.

As usual, I reserve the right to be wrong.



Comments Disabled

Amazon’s Profit Picture - One Strike & It Is Out?

Eugen G Tarnow  August 10 2017 11:37:52 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

I knew that Amazon makes most of its money from its Amazon Web Services (AWS) and not from selling products online.  But I never really knew the numbers.  Here they are:



Amazon makes the bulk of its revenue from non-AWS sources but the most of the profit comes from AWS and the profit margin is all AWS.  In fact, the profit margin on non-AWS revenue is only 0.9%, about the same as for a supermarket.

The acquisition of Whole Foods with a profit margin of about 3% then makes Amazon less vulnerable to a downturn. But is seems that if there is ever a long strike at Amazon, it might just kill the business. As a comparison, the profit margin of Alphabet is 16%, though declining 4% per year.

But Amazon also may be an example of why worker salaries are increasing so slowly: There just is no room for salary increases at Amazon and since they are killing the competition, neither is there room for salary increases elsewhere in retail.

The shareholders are presumably hoping that Amazon will kill its competition and then raise the prices and make a real profit. But if it raises the prices on the customers it may open the door to anti-trust action...

Numbers are taken from https://revenuesandprofits.com/amazon-revenues-profits-analysis-2017-update/

As always I reserve the right to be wrong.
Comments Disabled

Amazon’s Reverse Machine Learning Algorithm Annoys Customer

Eugen G Tarnow  August 10 2017 09:27:24 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

Machine Learning is still in its infancy at Amazon - at least when it comes to customer support.  

I had ordered an Amazon certified lightning cord for my iPhone and never received it.  There was no way for me to communicate this issue - the only complaint one could file was to return the item.  Since I never received the item I could not return it.  

This already raised my eyebrows: The quality of the Amazon customer support software is certainly below par if it does not allow the user to indicate that an item was never received.

I decided to put up a review (below) and see if that would work.

The Amazon machine learning algorithm rejected my review and foisted the responsibility of figuring out why it was rejected (I term this "Reverse Machine Learning") onto me, the customer.   Presumably it had to do with the ALL CAPS subject line.  But then it should have just said so.  And that rule would be too simple since some old people use just CAPS.

Not a very attractive use of machine learning and reversing it on the customer - not a good idea.

Amazon machine learning email:

Your review could not be posted. Thanks for submitting a customer review on Amazon. Your review could not be posted to the website in its current form. While we appreciate your time and comments, reviews must adhere to the following guidelines: http://www.amazon.com/review-guidelines

from Something on August 9, 2017 NEVER RECEIVES SOMETHING AND THERE IS NO WAY TO COMPLAIN I never received it and there is no way for me to communicate with Amazon to tell them

We encourage you to revise your review and submit it again. A few common issues to keep in mind: Your review should focus on specific features of the product and your experience with it. Feedback on the seller or your shipment experience should be provided at www.amazon.com/feedback. We do not allow profane or obscene content. This applies to adult products too. Advertisements, promotional material or repeated posts that make the same point excessively are considered spam. Please do not include URLs external to Amazon or personally identifiable content in your review.

This message is for: etarnow@avabiz.com. If you'd rather not receive future e-mails of this sort from Amazon.com, please opt-out here.

Reference ATVPDKIKX0DER-R21PPOAHIVZ9MG. (c) 2014 Amazon.com, Inc. or its affiliates. Amazon.com, 410 Terry Avenue N., Seattle, WA 98109-5210.

Comments Disabled

Correlation between Medicaid Expansion And Overdose Deaths

Eugen G Tarnow  July 18 2017 08:01:05 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

Some states chose to expand Medicaid under ObamaCare and some did not.  Those who chose to expand Medicaid tend to have more deaths from overdoses (28% more, p<0.03).




It is unclear why this relationship holds.
Comments Disabled

Google Does Not Scale

Eugen G Tarnow  June 28 2017 07:15:54 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

In the aftermath of the EU decision against Google's manipulation of its search results to favor its own products, it is instructive to ask why Google did what it did.

The answer is, perhaps surprising: Google had to do it because Google does not scale. There are too many web pages and Google can't figure out what the order of the search results should be, including where its own products should display.

Let me repeat it.  The Google search algorithm does not scale.  

If it did scale, Google would not have to change its algorithm all the time.

The reason it does not scale is that Google does not have enough statistics of what search results we prefer to click on.  The reason for that is that we rarely go beyond the first 5-10 results, making results further down the list for all purposes nonsense to Google.  Google just has no idea whether search result 15 should be higher up than result 16 because neither get enough clicks.  And Google has really no idea whatsoever once we get to search result 301.  

Thus Google functions sort of as an intelligent phone book with 10 results that may be relevant.  The rest might as well never show up.

If Google wants to sell its own products and the relevant search result starts out as 301, it will never move up.  So Google has to cheat and move it up to bypass its own search algorithm which does not scale.

Here is an example.  I wanted to know who purports to measure the time it takes to fill out government forms.  My search query was "who measures how long it takes to fill out a government form".  The 2nd result was "How the Government Measures Unemployment".  The 8th result was "Handbook of Sexuality-Related Measures".  I did not find the answer I was looking for.

So Google is great for answering some popular questions, to find the five largest manufacturers of a particular gadget. and to direct users to the businesses in Google's business directory (as long as they get to push their own directory ahead of everything else).

But if you are a small business trying to sell something you don't have a chance.  And don't let the SEO companies tell you otherwise.  

Google makes everybody scurry around in the false belief they can show up on the first search page.  It changes its algorithm and makes everybody run around again.

It is just a waste of time - because Google does not scale.

Added September 1, 2017: Google does not seem to know how to alphabetize.  A search on Big Little Lies indicates that the first choice of outlet starts with a "Y":
.

Added September 25, 2017.  When searching for a drug covered earlier in this blog ( http://mitphd.com/Check%20Big%20Data%20Blog.nsf/dx/an-unpopular-drug ), Google puts the uncorrelated drugs.com on top (see http://mitphd.com/Check%20Big%20Data%20Blog.nsf/dx/internet-drug-ratings-askapatient.com-webmd-correlate-well-but-drugs.com-does-not ):


Comments Disabled

A new algorithm for promotion optimization?

Eugen G Tarnow  June 26 2017 06:31:52 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

We seem to have discovered a new algorithm for promotion optimization.  

Algorithms are not patentable so we keep them proprietary.  

But if anyone is interested in comparing their algorithm to ours, we will be happy to participate.  



Comments Disabled

Getting 3% growth by increasing the workforce: have fewer people go to college and abbreviate high school

Eugen G Tarnow  June 25 2017 09:07:15 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

Rex Nutting of MarketWatch wrote an opinion piece titled "Sorry, Mr. Trump, but the only way to get to 3% growth is to hire more Mexicans ".

He stated that in order to get that 3% growth we would have to hire another 33 million workers of the next ten years.

We can get there half-way by having companies stop insisting on college credentials that are not needed in the workplace.

Today 20 million people are attending college.  If we cease being a "credentialist" society, and, over ten years, can get say 75% of college students working instead of studying, then we will get 15 million workers into the workplace.

And think of how many fewer people would be burdened by those student loans...

Another 25% of what is needed can be had by abbreviating high school (there are 15 million in high school at any one time). Much of what is studied there is not needed in the workplace either.  Cut two years off of high school and another 4 million workers can be added.

As always, I reserve the right to be wrong...




Comments Disabled

Earnings correlate less and less with cognitive ability - at least for Finnish men - or is it just Nokia?

Eugen G Tarnow  June 25 2017 06:59:47 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

Evidence just arrived showing that earnings at 30 are correlating less and less with cognitive ability. At least among Finish men since the year 2000.  (See Jokela, M., Pekkarinen, T., Sarvimäki, M., Terviö, M., & Uusitalo, R. (2017). Secular rise in economically valuable personality traits. Proceedings of the National Academy of Sciences, 201609994.)

Finnish men were tested for their personality traits and their cognitive abilities as they were ready to join the armed forces draft.  Below is displayed the correlations within each cohort of the test results with earnings at 30.  



So right around the tech bubble, the correlation of cognitive ability with pay peaked.

But wait.  Finland is known to be a one-company country (see http://www.economist.com/node/21560867 ).  This is what happened to Nokia stock around the tech bubble:



Could it be that the earnings findings just pick up what is going on at Nokia?  


Comments Disabled

WSJ’s Daily Shot covered our research

Eugen G Tarnow  June 22 2017 09:49:37 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

The Wall Street Journal's Daily Shot covered our research.  See https://blogs.wsj.com/dailyshot/2017/06/22/wsjs-daily-shot-us-shale-oil-output-to-hit-record-high-in-july/ and scroll to "Avalon".


Comments Disabled

Colleges improve critical thinking only up to the 1250 score on the CLA+

Eugen G Tarnow  June 5 2017 04:17:45 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

The Wall Street Journal published the CLA+ scores of 68 colleges. CLA+ is a test to measure the ability to think critically.  

I graphed the score improvements as a function of the incoming class score.  It appears that colleges with low scoring incoming classes are able to increase their score but this increase disappears as the students get smarter.  

At 1250 colleges no longer improve the score averages.


Comments Disabled

Facebook - a drag on productivity?

Eugen G Tarnow  June 2 2017 09:42:24 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

The New York Times wrote that "Facebook Has 50 Minutes of Your Time Each Day. It Wants More.".  

That translates into 100 hours a quarter.  

Facebook is able to make $20 per US user for those 100 hours.  

That translates into $0.20 per hour.  

Suppose the advertisers spend 10% of their revenue marketing.  If Facebook ads are of average utility then the advertisers make $2 per hour.  One hour of your Facebook time is then creating $2.20 value for others.

Thus an hour worth $20 is transformed into an hour worth just $2.20:



Data scientists and data engineers working for Facebook attempt to keep your attention within Facebook.  

That also means they are trying to remove 89% of the value you could derive from your time.

The Zuckerberg proposal of a "Universal Basic Income" seems to be in line with this - the UBI would remove much of the competition for attention allowing Facebook to get more user time.

What did we forget?  Can the consumer surplus save the equation?  If would seem that we would need a very large consumer surplus.

It seems that this is a relatively new issue.  Video games and Netflix binge watching seems to fall in the same category - both are designed to keep the user from leaving and probably do not make much money per hour either.

As always, I reserve the right to be wrong.




 
Comments Disabled

CEO pay for S & P 500 firms: Women paid same as men & have the same one-year return. Still no correlation between pay and share holder value.

Eugen G Tarnow  May 31 2017 10:32:59 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

The Wall Street Journal just published the CEO pay and share holder return for the S & P 500 companies.  Their conclusion (women CEOs are paid more) is different from mine: the women are not paid more than the men - they are paid the same, at least if one adjusts for statistical noise.  

Neither gender is better for share holders.

Here is the result of my ANOVA:

Field	Men	Women	p (useful if <0.05)
Total CEO Pay ($million)	13.288	14.346	0.65
ShareHolderReturn (%)	0.143	0.232	0.10

And here are the graphs:




As usual, there is no correlation between CEO pay and share holder return:








Comments Disabled

The Best New Jersey Soccer Clubs

Eugen G Tarnow  May 18 2017 09:22:14 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

Which are the best kids' soccer clubs in New Jersey?  Some clubs have a great 2003 team and a terrible 2002 team - can you even compare them?

Perhaps it is just chance?

It turns out it is not chance.  I compared the clubs that had the ten highest nationally ranked 2003 teams out of which I kept those with at least four nationally ranked teams in different age groups.  I performed ANOVAs on the inverse ranking and on the log(ranking) and found that, indeed, there is a statistically significant difference in both cases (p<0.0008).

The club rankings come out slightly differently depending upon how you do the averaging.  On top is displayed the inverse of the inverse average and on bottom is displayed the exponential of the log averages.  

The winner is Match Fit, second place is taken by TSF and third place is shared by FC Copa and Morris United..




Comments Disabled

When calling persistence pays off

Eugen G Tarnow  May 9 2017 10:30:52 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

When making sales calls persistence pays off.  Here is what happens when bank sales people call potential customers to get a term deposit.


Comments Disabled

Attractiveness or a barrier?

Eugen G Tarnow  May 3 2017 12:17:52 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

Louis Vuitton's VP of Analytics, Carey Chou, performed a fascinating and thorough analysis of their web sales data and found that some products act as attractors to other products.  See this example (taken from https://sharedmemory.github.io/blog/2016/10/13/customer-choice-model-and-attraction-effect/ ):



The probability that the customer will buy a particular bag is changed as a third alternative is introduced.  This third alternative makes the first bag more attractive and the second bag less attractive.  Note that the probability of sale is extremely low for this third bag.

But perhaps the third bag acts somewhat as a barrier preventing the user from scanning to the right?  The effects are small but the ideas are interesting.
Comments Disabled

Facebook, MIT, Gender & Rejection

Eugen G Tarnow  May 3 2017 05:36:07 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com



Source:
https://www.wsj.com/articles/facebooks-female-engineers-claim-gender-bias-1493737116
http://www.collegeconfidential.com/dean/000255/

10/30/18: There is more data in MIT's Common Data Set (see https://web.mit.edu/ir/cds/index.html )

:
Probability of being admitted to MIT once you apply.

NOTE: the applicant pool is manipulated by MIT (as is the applicant pool of other colleges such as Harvard) encouraging selected students. That presumably means that without the manipulation there would have been fewer women applicants and thus the disparity between men and women is in reality higher than the graph indicates.

MIT does not release test scores by gender so they are presumably considerably lower for the women.


Comments Disabled

GNP seasonality adjustments appear to be just fine

Eugen G Tarnow  April 30 2017 09:34:17 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

The Wall Street Journal questioned the GDP seasonal adjustment methods as the latest quarter came in at a low 0.7% growth and it seemed like the first quarter often comes in low:



An ANOVA test of their hypothesis, however, shows that there is no significant difference between the quarters (p=0.18) so the seasonal adjustment appears to work just fine.  

(I read the numbers off of the graph as:

1st quarter	2nd quarter	3rd quarter	4th quarter	Year
	3.75	2.75	2.45	2010
-1.5	3.25	0.75	4.5	2011
2.75	1.875	0.5	0.1	2012
2.8	0.8	3.1	4	2013
-1	4	5	2.2	2014
2	2.65	2	0.85	2015
0.8	1.4	3.5	2.1	2016
0.7				2017

).
Comments Disabled

"Purdue Pharma will partner with the Commonwealth of Virginia to enhance its prescription monitoring program to address opioid abuse."

Eugen G Tarnow  April 18 2017 03:58:01 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

Purdue Pharma is one of the largest opioid distributors.  

I guess that means Purdue Pharma has the Commonwealth of Virginia in its pocket.  See http://healthitanalytics.com/news/prescription-monitoring-program-to-reduce-opioid-abuse-in-va
Comments Disabled

A Perfect Storm - US Health Care

Eugen G Tarnow  April 10 2017 12:02:11 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

Vanguard just sent out a newsletter that identifies "the structural forces dampening global growth and interest rates".  

They are:

• Technology - falling technology costs
• Demographics - an aging populace
• Globalization - competition on prices and labor

Thus far health care has been immune to, or benefited from, all three forces.

Health care technology costs are typically increasing, not decreasing, and used not to eliminate inefficiencies but to open up new product lines.  Since the 1970s a substantial number of doctors could have been replaced by computers since AI systems were already then giving good enough diagnoses.  AI diagnostic systems have improved but their use is always limited by some kind of licensing requirement. At the same time, electronic medical records have become more and more expensive and cumbersome.  They do not include a diagnostic field presumably to prevent AI from taking over.

The demographics will increase the need for medical care. After Medicare was introduced, doctor salaries increased from three times the average salary to seven times ( http://www.medpagetoday.com/Columns/At-Large/27158 ).  The demographic change will, no doubt, bring this inequality to new heights.

Health care has been able to escape globalization by a variety of protective laws.  Drugs cannot be imported, FDA prevents European approved drugs from being used, doctors are very hard to import (unlike computer programmers) and that is restricted also by state lines...

That adds up to the perfect storm.  While most families in the US have had their salary growth stop due to technology and globalization, health care has not and demographic changes will exasperate this.

Health care will give us the perfect storm.


Figure from http://www.mnhospitals.org/Portals/0/Documents/education/7016_GS01.pdf
Comments Disabled

Is there something fishy with Johns Hopkins University’s Pima Indians Diabetes Data Set?

Eugen G Tarnow  April 7 2017 11:43:47 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

This is a famous data set describing the incidence of diabetes in a population prone to diabetes.  It can be downloaded from here: https://archive.ics.uci.edu/ml/datasets/Pima+Indians+Diabetes .  

Some thirty data science publication have resulted.  But there are some strange things going on in this dataset.  

First, the age distribution of the participants is exponential:



Second, the body mass index does not increase with age:



I wrote the dataset depositor but did not receive an answer.  I wrote the archivists at University of California Irvine and they decided to just leave the dataset up.

But it seems there is something very wrong with it.  An the publications that resulted - are they therefore wrong too?

As always, I reserve the right to be wrong.

Comments Disabled

Is Restless Leg Syndrom based on a flawed survey?

Eugen G Tarnow  April 6 2017 02:25:21 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

When one constructs a survey it is important that the questions are not leading to the answer.  

This is not the case with the "International Restless Leg Syndrome Rating Scale" - see http://www.rls.org.au/pdf/PKGD6.pdf ).

Each question suggests to the patient that they have Restless Leg Syndrome (RLS).  For example:

"1. Overall, how would you rate the RLS discomfort in you legs or arms?"

"2. Overall, how would you rate the need to move around because of your RLS symptoms?"

etc.

This survey was produced by "The International Restless Legs Syndrome Study Group" and "validated" in a journal called SleepMedicine (see http://www.sleep-journal.com/article/S1389-9457(02)00258-7/abstract ).

My guess is that there is a substantial overlap between high scores on the International Restless Leg Syndrome Rating Scale and and suggestibility.

No wonder there are a lot of people who perceive themselves to suffer from RLS.

No wonder there is no known cause of RLS.

And no wonder that the placebo effect is so large: a-clinical-trial-with-humorous-results

But seriously, how much is this costing us tax payers?
Comments Disabled

A clinical trial with humorous results

Eugen G Tarnow  March 31 2017 03:43:44 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

We humans exist on a normal distribution and it is the job of healthcare marketers to keep chipping away at the tails.  

One such effort is the "Restless Leg Syndrome".  So GlaxoSmithKline, the National Institute of Health, the FDA and somebody else whose logo I cannot identify



set out to perform a clinical trial.  They used the "International Restless Legs Syndrome Rating Scale":

http://www.rls.org.au/pdf/PKGD6.pdf

The maximum score is 40.  It does not say in the clinical trial results what the baseline scores were.

https://www.clinicaltrialsregister.eu/ctr-search/rest/download/result/attachment/2005-005372-32/1/2034

The results were overwhelming:



As soon as the treatment period started, RLS patients started to improve!  The improvement kept increasing until about 8 weeks into the trial where it stopped at an impressive 12 points with the placebo and 15 points with the drug!

The upshot is then that anybody with RLS should ask a friend to start a "clinical trial" on the sufferer - and her/his leg will stay put.

Comments Disabled

Is the opioid epidemic a rural issue?

Eugen G Tarnow  March 29 2017 01:10:33 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

It was suggested to me that the opioid epidemic may concentrated in the rural areas of the US.  Interestingly, there is no correlation between Medicare opioid claims in a state and how rural the population is:



There is also no correlation between the rate of overdose deaths and how rural the population is:


Comments Disabled

A Press Release on Opioids - Record Number of New Jersey Doctors Disciplined in 2016 As State Ramps up Efforts to Curb Opioid Addiction

Eugen G Tarnow  March 21 2017 10:28:23 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

The New Jersey Department of the Attorney General issued a press release earlier this month about 31 medical doctors disciplined for opioid issues (you have to ask for the actual 31 names, they are not in the press release but you can find them on the bottom of this blog). It states "Purging the medical community of over-prescribers is as important to our cause as busting heroin rings and locking up drug kingpins.”

Well, if that is the case, few kingpins would be locked up and then for only a short time.

All but 5 of the doctors had prescribed opioids within Medicare.  The average number of Medicare prescriptions for these doctors were 650 per year.  327 NJ doctors had prescribed more than that within Medicare which suggests a conviction rate of 10%.

The majority of opioid sanctions were temporary: 61%.

The doctors sanctioned for opioids have somewhat more sanctions than the average sanctioned doctor (1.9 versus 1.5).  One doctor had 8 sanctions altogether - 7 previous sanctions.

That the state is indeed ramping up its efforts is shown by a graph of the number of doctors sanctioned by month covered by the press release - it is increasing:


However, the total number of sanctions by the medical board is the same over the last ten years, about 114 (which probably works out to about 0.4% of all active physicians in NJ):


There were also personal tragedies among the doctors - at least one had a child who overdosed on the doctor's opioid pills.

Here is an interesting question: will anybody sue these 31 doctors and be able to collect from their malpractice insurance?  Will the insurance companies then represent these doctors?

Many of the doctors had multiple versions of their names, one even had the first and last names switched in the different data sets.  Furthermore, not all sanctions could be found on the website of the State Board of the Medical Examiners (http://www.njconsumeraffairs.gov/bme/Pages/actions.aspx). One wonders whether data quality issues make it harder to catch perpetrators.

Final 2016 CDS Case List.docx
Comments Disabled

The Art of the Deal: Sweden pays the least for drugs in all of Europe (and much much less than the US)

Eugen G Tarnow  February 24 2017 03:06:02 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

When looking at Sweden, there are many gems to be found.  One is how they negotiate their drug prices.  It is something that we all should take a look at.

An Epipen is $40 and nobody forces you to buy 2 of them.  In the US a pair is $700 and you have to buy both.

See: http://www.tlv.se/Upload/Ovrigt/Rapport_internationell_prisjamforelse_lakemedel_151215.pdf




For prices of particular drugs, see: http://www.tlv.se/beslut/sok/lakemedel/

Call your politicians and your journalist friends and ask them to cover this.


Comments Disabled

Algebra 1 Scores & Social Group: Differences Increase from 2015 to 2016

Eugen G Tarnow  February 21 2017 01:43:24 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

As income inequality increases, one would expect standardized test scores to do the same.  In the NJ PARCC Alegra 1 test, this is indeed the case.  Here are the 2015 scores by social group:



And here are the differences between 2016 and 2015



As the social group increases from A to J, the scores go up.  This corresponds to 73% of the variation:



As always, I reserve the right to be wrong.
Comments Disabled

H1B Salary Distributions

Eugen Tarnow  February 2 2017 11:45:46 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

The salary distributions of H1B workers - for single applications looks like this.  The most common salary is $60-65,000.

For applications with two workers needed at the same time the distribution narrows and, again, the most common salary is $60-65,000 per year.

In Silicon Valley the distribution for single applications is centered around $100,000:


For applications with two workers it is much lower with the most common salary between $75-80,000:


In New York City, the single application distribution is as follows:



and the two-worker application distribution is:



The most common salary in the second case is $70-75,000.  

Cognizant typically applies for 30 workers at a time.  For those applications the salary distributions look like:


The most common salary is $60-65,000.

The data was taken from https://www.foreignlaborcert.doleta.gov/performancedata.cfm using fiscal year 2015 and when the employer gives a range in pay, the lowest number was chosen.
Comments Disabled

Which Occupations Need The Cheapest H1B Salaried Workers?

Eugen G Tarnow  January 31 2017 01:56:59 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

One might be curious which occupations the US just don't have enough workers in but at the same time are so easy to find abroad that they can be paid the minimum allowable H1B salary? The majority is programmers and analysts - no need to focus on the physical therapists (???).



The data was taken from https://www.foreignlaborcert.doleta.gov/performancedata.cfm using fiscal year 2015 and when the employer gives a range in pay, the lowest number was chosen.
Comments Disabled

Which US Counties Have No Smart STEM People & Need The Most H1B Workers At Minimum Allowable Salary?

Eugen Tarnow  January 31 2017 01:44:13 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

Which counties just don't have enough smart workers so they have to hire minimum salary H1B workers?  Here is a list and it includes places with the lowest STEM ability like Silicon Valley, New York, King (where Microsoft is), Dallas, San Francisco, Bergen (where I live).  



The data was taken from https://www.foreignlaborcert.doleta.gov/performancedata.cfm using fiscal year 2015 and when the employer gives a range in pay, the lowest number was chosen.
Comments Disabled

Which Attorneys File The Most Applications Paying H1B Workers the Legal Minimum?

Eugen G Tarnow  January 31 2017 01:30:16 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

One might be curious who are the lawyers that file the most H1B applications with the minimum legal limit?  See below.

The data was taken from https://www.foreignlaborcert.doleta.gov/performancedata.cfm using fiscal year 2015 and when the employer gives a range in pay, the lowest number was chosen.





Comments Disabled

Which Companies Pay Less Than 1% Over Legal Minimum for H1B Workers?

Eugen Tarnow  January 31 2017 01:11:46 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

Which companies pay their H1B workers the very minimum?  Here is a list.  The largest number of minimum wage applications come from Indian companies.  But we have American ones as well:  PWC, Deloitte, Accenture, CapGemini, Apple, Oracle, Cisco.



Do you think our colleges are any different?  Here is a list of the colleges with the most workers with minimum pay:



The data was taken from https://www.foreignlaborcert.doleta.gov/performancedata.cfm using fiscal year 2015 and when the employer gives a range in pay, the lowest number was chosen.
Comments Disabled

Statistics of the H1B Program: Most H1B Workers Get Minimum Pay

Eugen G Tarnow  January 31 2017 12:18:36 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

The H1B program is a goldmine for the corporations participating.  While it is supposed to supplant the lack of STEM talent in the US, the evidence is not there.  

Here is the promised salaries for single and multiple H1B applications.  Note that 44% (72%) of single (multiple) applications pay less than 1% over the prevailing wage, which is the legal minimum pay.

The data was taken from https://www.foreignlaborcert.doleta.gov/performancedata.cfm using fiscal year 2015.




Comments Disabled

School District Income Seems to Mean Little to the Poor

Eugen G Tarnow  January 25 2017 06:50:15 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

I had a conversation with a school board member in a New Jersey district in which the mean family income is large but economically disadvantaged students do not do well.  I decided to see if this was a larger phenomenon.

Indeed it was.  For example, below is shown the grade 6 PARCC math scores as a function of district family income for all children (top) and for economically disadvantaged children (bottom).  There is a strong correlation between the math scores and the district family income in general (R square is 44%) but not for the poor (R square is only 2%).  

It is also interesting just how expensive it is to increase the average grade by 1 point: $5,000 in mean income!


Comments Disabled

Student Loans Go Missing

Eugen G Tarnow  January 19 2017 12:59:59 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

Student loans are in the news because the Department of Education had a programming glitch that understated the number of delinquent student loans.

It is interesting to see how the student loans become delinquent.  I used the data from https://studentaid.ed.gov/sa/about/data-center/student/portfolio .

Student loans that are 31-90 days delinquent have a 56% chance of becoming delinquent 91-180 days, then a 57% chance of becoming delinquent 181-270 days and then a 70% chance of becoming delinquent 271-360 days.  

After that the loans are to be transferred into collection.  The number of loans in collection is not reported, only the number of loans transferred into collection.

This is where it becomes interesting.  One would think that the loans would go into collection as soon as they reach 361 days.  But after a quarter, there is no correlation between the delinquencies and the collections (all in billions):



Rather, if we plot the linear coefficient between delinquencies and collections as a function of the time delay, we find that it takes 6 quarters before there is a relationship between the two and then the relationship is a negative one, the more delinquencies, the fewer of them go to collection:



If we plot the missing loans as a function of quarter we find that it keeps rising:



If we add up the total missing loans after so many quarters we get:



Forecasting by extrapolation we get:



It will reach $1 Trillion in 68 quarters from now, in 17 years.  

As always, I reserve the right to be wrong.
Comments Disabled

NJ PARCC Algebra 1 Scores Changes Over A Year

Eugen G Tarnow  January 4 2017 06:49:41 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

The 2016 PARCC scores are out and they are quite interesting.  The zero score is 650 and the highest possible is 850, a 200 point spread. Below I show you the 2016 scores versus the 2015 scores in Algebra 1 for all schools.  There is a spread - the scores of those two years are not quite the same.



If I plot the difference between the two years as a function of the 2015 score we end up with the plot below.  Overall the average scores have improved by 5, the standard deviation is 9, and the improvement is highest for the lowest performing schools.  Some schools improved by as much as 50 points but some schools declined by 40 points.  



Some of these changes are statistical noise as can be seen in the corresponding plots of schools with at least 100 valid scores.  Again the improvement is on average 5 points but the standard deviation is a smaller 7.6.  The improvement is highest for the lowest performing schools.  Some schools improved by 35 points and some declined by 20 points.



What could account for the large differences?  The students or the teachers?  Let's increase the number of valid scores to 200 to lower the noise from the student population further.  The corresponding plots are below. The improvement is on average 5 points and the standard deviation is 7.3 and the improvement is highest for the lowest performing schools.  Again some schools improved by as much as 20-30 points and some declined by 10-20 points.



Since we varied the number of students and got similar results, it would seem that the difference in test scores are due to the teachers. In other words, variability between Algebra 1 teachers within the same school account for about 7 PARCC points (but can vary between -20 to 35 according to the plot just above).

Finally, the extra 5 points are somewhat dependent on socioeconomic characteristics, see the plot below.  "A", "B" etc. refer to "District Factor Groups" and the order is low to high socioeconomic characteristics.



The score differences as a function of socioeconomic characteristics are shown below:


There is no strong relationship of large declines or improvements with socioeconomic characteristics, see below:



As always, I reserve the right to be wrong.
Comments Disabled

NJ PARCC English 11 Scores Changes Over A Year: Differences Due to Socioeconomic Factors Minimized by Excluding Elite Students

Eugen G Tarnow  January 4 2017 06:49:41 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

On September 9th, 2015 Commissioner Hespe allowed 11th grade students who were enrolled in AP or IB courses AND took the assessment associated with those courses a “waiver” from the PARCC grade 11 ELA assessment.

The consequences of excluding the elite students from the PARCC test were large and strongly correlated with socioeconomic characteristics, see the plot below.  "A", "B" etc. refer to "District Factor Groups" and the order is low to high socioeconomic characteristics.  



The waiver decreased the scores of high performing schools and increased the scores of low performing schools.  The score differences as a function of socioeconomic characteristics are shown below:



The consequence can be seen the clearest in the graph below that shows the average scores by socioeconomic group for the two years plotted against each other.  The score differences in 2016 are only 44% of the differences in 2015.  It looks as if socioeconomic factors are only 44% of what they were before.



Another way to put it: the F test for socioeconomic characteristics yielded 29 in 2015 but only 14 in 2016.

As always, I reserve the right to be wrong.
Comments Disabled

Providers in Republican States Prescribe Higher Levels of Opiates Than in Democratic States

Eugen G Tarnow  December 6 2016 09:13:23 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com


AP wrote a story ( click here ) about how the makers of opioids lobby state legislatures very successfully.

I found independently that Medicare opioid prescriptions are very strongly correlated with state poverty - the poorer the state, the more Medicare opioid prescriptions.  This presumably means that the state legislatures in poor states are easier to pay off than state legislatures in rich states.  The payoff results in opioid favorable state laws; a lack of laws limiting opioids.

I then decided to look at whether there is a difference between parties: do states with Republican legislatures write less opioids than states with Democratic legislatures or vice versa?  

It turns out that states with Republican legislatures have providers that write more Medicare opioid prescriptions. This is shown in the graph below.  In each of the graphs is displayed the percent Medicare opioid claims out of all Medicare claims as a function of median household income.  The red dots correspond to Republican governors and the blue dots to Democratic governors.  The lines are the least square fits to the dots.  

In the top graph I find that states with Republican governors have more Medicare opioid claims than states with Democratic governors. In the middle graph we see that is even more so for Republican upper houses and, in the bottom graph, for Republican lower houses (I used the Wikipedia definition of lower and upper houses).



This result is not obvious.  Below are the same graphs for the murder rate.  It is a lot less sensitive to the median household income and the variation with party is not as consistent.



A one-way ANOVA showed that there was a significant difference between Democrat and Republican Lower Houses for opioids (F=16.7, p<0.001) but not for the murder rate (p=0.8), Upper Houses for opioids (F=4.6, p=0.015) but not for the murder rate (p=0.67), Governor for opioids (F=4.23, p=0.045) but not for the murder rate (p=0.58).
Comments Disabled

Do Math Supervisors Influence PARCC Scores?

Eugen G Tarnow  November 19 2016 09:05:16 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

I have previously written about how difficult it is for a district to get good math teachers (as reflected in the weak correlation between district parental income and PARCC scores).  

This time I asked the question: do math supervisors matter for the PARCC scores?

If they do, then if the math supervisor is good, all the math scores would tend to be good and if the math supervisor is bad, all the math scores would tend to be bad.  In other words, there should be a strong correlation between the math scores of a school district if the math supervisor matters.  If the math supervisor does not matter, the math scores would be correlated with each other no more than with some other subject.

I compared the correlations within the PARCC math scores with the correlations between the PARCC math scores and the NJBCT biology score.  I only included schools with complete scores, see http://njschoolperformance.com.

The result was: no statistical difference between the two.  In other words, biology is as correlated with the math scores as math scores are correlated with each other.

Thus it seems that math supervisors do not influence the PARCC scores.
Comments Disabled

Globalization helps unfree nations become free but free nations move the other way

Eugen G Tarnow  September 21 2016 09:05:52 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

Here are two charts I find interesting in the midst of the discussion of globalization.  I used the Index of Economic Freedom (Heritage Foundation and the Wall Street Journal) to see what globalization does to free nations.  The left chart shows that GDP per capita tends to correlate strongly with freedom.  BUT the right chart shows that during the last three years, freedom did not correlate with the GDP ratio.  So at least in one measure globalization is not favoring free countries.



The average freedom score, weighted by population, has increased by 0.7 points in the same time period, a positive sign.  However, freedom is moving from free to unfree countries: the difference in freedom correlates negatively with the freedom score.



Something to discuss at this week's meeting of the United Nations General Assembly, no?

And here is how we in the United States have lost our freedom since 2009:


 
Comments Disabled

The eight best school districts of NJ - a district for every budget

Eugen G Tarnow  September 13 2016 02:09:05 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

When a family is looking to settle down in NJ they are often trying to figure out what is the best school district they can afford.  

We ranked all the school districts based on their PARCC test data (http://NJSchoolPerformance.com ), including only school districts with complete data.  

For every income level there is a highest ranking school district and here is our list:

School district	Avalon ranking	Median family salary
PHILLIPSBURG TOWN (Warren County)	62	$52,484
JERSEY CITY (Hudson County)	54	$64,419
FORT LEE (Bergen County)	46	$87,289
NEW MILFORD (Bergen County)	17	$97,639
LEONIA (Bergen County)	14	$100,283
EDISON (Middlesex County)	12	$105,670
MAHWAH (Bergen County)	4	$120,107
BERNARDS (Somerset County)	1	$165,714

Comments Disabled

Educational Quality Control Lacking - New Jersey School District Subject Knowledge Test Results & Family Income

Eugen G Tarnow  September 6 2016 09:58:15 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

We have seen that family income determines 45% of the Avalon school ranking of New Jersey school districts.  

But income does not determine specific subject knowledge.  In other words, you do not get the bang for the buck you might expect in each subject.  

In Figure 1 I show you how much of the subject knowledge is determined by the family income in NJ school districts.  Math knowledge increases from Algebra 1 to Algebra 2 perhaps because the Algebra 1 score is influenced by both middle school and high school students which might make the teachers difficult to supervise.  The trend is the opposite in English, presumably because parents are strongly involved in English 9 and then give up their involvement resulting in an almost random English score in grade 11.

In other words, while district family income determines the AVERAGE score over all subjects, the SPECIFIC scores are left to chance.

If you think school is important, it is time to get involved.  A talk with a low performing department supervisor might be a good beginning.


Figure 1.  Importance of district income in determining subject knowledge.
Comments Disabled

New Jersey School Districts Rank Determined By Family Income But Bargains Still Remain to Be Had

Eugen G Tarnow  September 5 2016 01:05:49 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

I performed probably the most comprehensive ranking of New Jersey school districts (see http://NJSchoolPerformance.com ).  Since SAT scores are determined by family income and less by the school performance, we ranked the New Jersey school districts based on the standardized test PARCC high school scores and the corresponding scores from middle school honors programs.

I expected there to still be a correlation with family income but I was taken back when I found that family income made up 45% of the ranking (see figure below).

Nevertheless, the remaining spread is large enough for there to be bargains to be had.  If you want to live in one of the top ten ranked school districts, expect to have to make about $103,000-$211,000 per year. (NOTE: the family income does not necessarily determine the amount of money the school get to spend per student - that's a different topic.)

And if you live in school district that has low ranking despite being in a town with high family incomes - talk to the corresponding board of education (and good luck with that!).


Figure. Avalon ranking of school district versus median family income.
Comments Disabled

NJ School Districts: SAT Scores, PARCC Scores, Suspension Rates & Family Income

Eugen G Tarnow  August 26 2016 10:44:50 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

Many families live in particular New Jersey towns for the school districts and at least some select the school district based on the SAT scores. However, a higher SAT school district carries with it financial hardship.  One can make a more optimal financial decision if one could tease out what part of the SAT scores are based on school performance and what is based on individual family spending on SAT coaching – since the coaching can take place in any school district.
For this purpose, I am able to exploit a new standardized test in NJ public schools, the PARCC test, which is relatively opaque to parents.  It was designed with a lowest possible score of 650, presumably to avoid parental shock, is accompanied by frequent assurances to students that the PARCC test scores are “confidential”, the student reports are almost unreadable (including a demographically matched set of school districts that do not match the local area) and the schools do not discuss their performance reports.  This suggests that the PARCC results reflect the combination of the quality of the school education and the quality of the students and less so parent involvement.  The SAT score, on the other hand, alerts parents and worries students as it carries great importance in college admission, suggesting that the SAT score reflects, in addition to the quality of the school education and the quality of the students, more student effort and parental input.

One important part of the student effort and parental involvement to increase the SAT score is paid coaching.  It would seem reasonable that the access to paid coaching is income based.  The lowest incomes should correspond to little paid coaching while the highest incomes should correspond to the best paid coaching available. Thus if one removes the effects of the school environment via the PARCC scores, the remainder should reflect the effect of best paid coaching available (this reasoning is similar to that of Zwick & Green, 2007).

Fig. 1 displays the effect of family income on PARCC and SAT scores as well as the student teacher ratio and the student suspension rate.  The student teacher ratio has a very small effect and will be neglected.  Income differences affect some subject scores more than others.  Note the trend for English – as high school progresses the correlation with income decreases.  The correlation with income is also low for Algebra 1. The strongest PARCC test correlation with income is the overall average.  The student suspension rate is also strongly correlated with income.

Fig. 1 PARCC and SAT test score correlations with median family income.  

In Fig. 2 is displayed the correlation with income of the SAT score with the PARCC average and the student suspension rates removed separately and together via a linear regression.  In Table 1. is displayed the SAT score difference for incomes between $50,000 and $170,000. The unaltered difference is 187 and the difference with the effects of the PARCC average score and student suspension rates is 74.


Figure 2.  Correlation of SAT score with family income removing effects from suspensions and PARCC averages.

		Removing suspensions	Removing PARCC Average	Removing both PARCC Average and suspensions
SAT score difference 50,000-220,000 income	187 (out of 800)	119 (out of 800)	91 (out of 800)	74 (out of 800)

Table 1.  The difference in SAT scores for school districts with mean family incomes of $220,000 and $50,000.

In Fig. 3 is displayed the student suspension rate as a function of income.  The functional form is close to exponential which means that an extra $30,000 in income results in the halving of the suspension rate.

Figure 3.  The effect of family income on suspension rate is exponential; an extra $30,000 in family income lowers the suspension rate by 50%.

In Table 2 is displayed the financial correlations of an SAT point.

	Increase in tuition per year	Increase in earnings per year
1 point increase in SAT score	$36	$61

Table 2.  The consequence of one SAT point.




Discussion
Many people live in particular New Jersey towns for the school districts and at least some select the school district based on the SAT scores.  I find that each SAT point correlates with an increase in the mean family income of about $1,000.  On the other hand, 74 points out of 800 are presumably attributable the cost of SAT coaching.  If the most expensive SAT coaching is $7,400 this is a lot cheaper alternative at a one-time cost of $100 per SAT point.  

I find that college average SAT scores correlate with $61 additional yearly income per SAT point, partially offset by a tuition increase of $147 for a four year stay per SAT point, suggesting that SAT coaching is worthwhile (breakeven point occurs after four years) but a better neighborhood may not be worthwhile (Tarnow, 2015).

Interestingly, the tuition increase for an SAT point is suboptimal for the colleges: for every point increase in the SAT score, a college can expect an increase in the median family income of $3,000 (in NJ, it may be different in other states).  However, they only tap $147.

Colleges could use an “effective” SAT score with the role of family income removed but if they did, they would correspondingly compromise their financial needs.

Within the PARCC test I find large variations in how much the median family income influences the subject scores (Fig. 1).  The influence is the lowest for Algebra 1 and 11^th grade English. The reason is unclear though I venture to guess that it has to do with a lack of parental involvement: Algebra 1 is taught both in middle school and high school and may be harder to keep track of and 11^th grade English may be a topic hard to understand and/or control.  I note that New Jersey is not a state in which parents are particularly involved with the schools other than comparing school districts.

The suspension rate declines exponentially with median family income.  $30,000 halves the suspension rate.  The cost of moving from a “bad” neighborhood to a better one is extremely expensive.  Interestingly, teachers in the NJ school system can do this for free even though others can only do this at a substantial cost.  The freedom of teachers to move districts may also be a reason for low performing school districts staying low performing.  I suggest a solution to the problem: tax the contract benefit of moving to a richer school district at the difference in median family income between the two districts.
Comments Disabled

The Opium for the Poor Is – Opium. Providers in States with Low Income Prescribe High Levels of Opiates

Eugen G Tarnow  August 26 2016 11:00:10 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

The current opiate epidemic has many reasons: pharmaceutical marketing, naïve and/or unscrupulous doctors, availability of cheap and cleverly sold Mexican heroin and inappropriate policy changes to encourage the idea that pain was something curable with drugs that were not addictive (Quinones, 2015).

I find that another reason is lax supervision of Medicare providers.  I find that the Medicare opioid claims per Medicare recipient varies strongly by state.  It should not.  Medicare should have caught that but did not.  

Interestingly the states with the largest number of claims are the poorest - with 51% of the variance coming from the median household income.


Comments Disabled

NJ School Performance Reports are surprisingly interesting

Eugen G Tarnow  August 19 2016 04:40:48 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

Last February the 2014-2015 school performance reports came out.  Here is a comparison between the high schools of two neighboring towns in New Jersey, Fair Lawn and Glen Rock.

1.  Fair Lawn has not posted the report on its high school website for the last two years - the only link available is to the old 2012-2013 report which is announced as "now available".  Glen Rock does not post any link at all.
2.  In English language arts/literacy the percentage who met or exceeded expectation was 57% in Fair Lawn and 42% in Glen Rock.
3.  In Math the corresponding percentages were 43% (Fair Lawn) and 28% (Glen Rock).
4.  The participation rate was 85% in Fair Lawn and 78% in Glen Rock, presumably indicating that the true difference in English and Math scores between Fair Lawn and Glen Rock may be larger.
5.  Biology proficiency is 72% for Fair Lawn and 85% for Glen Rock.

Let's think about that for a minute.  If Fair Lawn is generally better than Glen Rock in both English and Math, why should the Biology score be better for Glen Rock?  There seems to be a problem with the Fair Lawn Biology program.

6.  The trend of meeting or exceeding expectations in English in Fair Lawn is downwards as students move from 9th to 11th grade: 60%, 54% and 36% in 11th grade.  Glen Rock has the same issue with a trend that goes 54%, 19% and 26%.

The conclusion is that there seems to be a serious problem with the high school English program in both schools.

7.  There is no such trend in Math, but the meeting or exceeding expectations in the Algebra 1 class of Glen Rock High School is a surprising 10%.  What is going on there?

8.  The average SAT score was 1602 for Fair Lawn and 1744 for Glen Rock.

Wait a minute.  Fair Lawn scores better in Math and English but when it comes to the SATs, Glen Rock wins hands down.  The SAT score seems to have nothing to do with the high school test scores!  Could it be that everyone in Glen Rock hires a tutor that is much better than its high school teachers?  Or do the Glen Rock students suddenly realize that they have to perform? How can Fair Lawn students copy that?
Comments Disabled

Big Data: If you can save 1% your are doing well - if you can save 30% run away! Response to Bloomberg’s Leonid Bershidsky

Eugen G Tarnow  June 2 2016 09:10:03 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

My experience in big data has taught me one piece of "big data philosophy": if you can save 1% or so, you are doing great.  If you see a way to save 30% - run away as quick as you can!

Before I explain the reason for this philosophy, let's use an editorial in Bloomberg by Leonid Bershidsky as background.  The title is "Big Data Is Still Only a Little Helpful and is quoting an article by Hyunyoung Choi and Hal Varian.

Lenoid writes that trends in Google queries can predict the number of current autosales before official numbers are made available.  In other words, because Google calculates auto query reports much faster than official reports are put together on actual sales, the Google data can be helpful.  The predictions can be improved by 5-20% according to Choi and Varian but they are looking at second order differences.  The forecast without Google may be 6.34% and with Google trends 5.66% giving an improvement of 10.6% but the actual difference is less than one percent.

If we consider the Choi and Varian improvement to be a 1% improvement, in my book they did a great job. People are pretty good at what they are doing and using a computer can make small improvements but it is hard to think of ways it can make a big difference on problems that have been studied many times over.

Or can it?

Sometimes it is easy to find much larger improvements using big data.  For example, I found that provider Medicare reimbursements are too high and they increase list prices by an average of 45%, that Medicare payments to hospitals for heart attacks could be lowered by 37% without effecting the quality of care.

Why didn't Leonid Bershidsky write about these results instead and change the title to "Big Data Can Save 40%"?

That is because if your big data algorithm finds such a large number it is likely it has encountered systemic corruption.

Finding 30% in fields such as health care is like shooting fish in a barrel.  But then run away unless you are big data's equivalent of Dirty Harry.
Comments Disabled

Who is more eloquent - conservatives or liberals? The grade level of the Facebook political discourse.

Eugen G Tarnow  May 18 2016 01:55:14 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

The Wall Street Journal recently devised a side-by-side comparison of what very conservative and very liberal users see when they go on Facebook.  

I thought it might be interesting to see whose discourse level was more sophisticated - conservatives or liberals?  

I used the comments that travel around with the news articles on all the topics categorized by the Wall Street Journal: "Hillary Clinton", "Donald Trump", "Bernie Sanders", "Barack Obama", "Guns", "Abortion", "ISIS", "Transgender" and "Facebook" and copied them into Word, replacing all names by "Jane" to remove any bias from "Donald Trump" having fewer characters than "Hillary Clinton", for example.

I then ran the Readability statistics and found:

Very conservative: 7.5

Very liberal: 6.6

using the Flesch-Kincaid Grade Level.

I did not check what the error bars might be (probably a full grade) but note that our high school education seems under utilized for both our very liberal and very conservative friends.

This entry is scored at the Flesch-Kincaid Grade Level 12.  

The texts used are displayed below:

Here's why Jane is APPEALING and ... Jane is APPALLING.
Brave Navy Seal DESTROYS Jane Right To Her Face.
This is awesome! We solute you soldier.
Can we get 5,000 'LIKES' for this Seal?
Jane's latest attack against Jane is a perfect example of why many Americans don't want her anywhere near the White House. This could be a BIG preview of Jane's strategy for the general election.
Will these kind of attacks help Jane defeat Jane?
Despite Jane's desperate attempts the facts of what really happened in the Benghazi attack are coming to light.
The New York Times's hit piece on Jane continues to unravel. Now Jane says she has nothing but respect for Jane and they lied about her account too.
?I want to promulgate the idea that just because someone has a different opinion than yours it does not make them the enemy,? Jane said.
"I?ve experienced more pure, unadulterated anti-Semitism since coming out against Jane?s candidacy than at any other time in my political career. Jane supporters...greeted the birth of my second child by calling for me, my wife, and two children to be thrown into a gas chamber."
Want nightmares?
WATCH:
He literally lost delegates with no one else running.
HEY SOCIALIST Jane FANS: CHECK OUT THE SHOCKING VIDEO OF SOCIALISM IN VENEZUELA
This family is not fit to run a lemonade stand let alone a country.
MSNBC?s Nicolle Wallace argued that if Jane loses to Jane in November it will be because Jane stayed in the presidential race so long and due to his effective attacks on the Democratic front-runner.
Jane supporters -- are you getting this?
"As the young American left continues to champion the rise of socialist policies in our own country, many are left wondering if these idealistic collectivists are really prepared for the realities of socialism." -- Generation Opportunity
SHOTS FIRED INTO Jane NEVADA HEADQUARTERS!
Public schools in Jane's hometown of Chicago have just taken this transgender bathroom insanity to the next level.
What they're demanding students do now shows exactly how set they are on brainwashing our children that perversion is perfectly normal.
This is sick, and it must be stopped NOW.
This is not OK!
MUSLIMS TAKING OVER A SMALL TOWN IN TENNESSEE In Jane's Plan For A "New America" [Video]
Uh...
What treachery. It it the greatest acts of treason by a President in American history (rivaled by the Benghazi betrayal and cover-up) and yet not a critical word in the press. Scant attention is paid to Jane's towering betrayal.
How many of you agree she should be headed for prison instead of the White House?
"People who really are struggling against oppression do not resort to inventing slurs or smelting smoking guns; they have enough to deal with in the real world."
Who could have ever predicted this would happen?
Jane was molested by male migrant gang in New Year attacks. The 17-year-old and her friend described it as 'the worst night of our lives.' She said worst part was witnesses, including police, did nothing to stop it.
These girls need guns just like the Jews needed guns -- Germany is reverting but this time the victims are Germans.
NYT & The Left are desperately trying to make Jane look like a cad.
Why don't they turn those guns of The Lolita Express's frequent flyer and serial rapist, Bill Jane?
ACLU Trying to Shut Down Catholic Hospitals Because They Refuse to Do Abortions
Yesterday's Supreme Court ruling on President Jane's abortion-pill mandate was a major victory not only for the Little Sisters of the Poor, but also for the religious freedom of every American.
Read and Share as we explain the Court's ruling and why it is significant.
Louisiana Legislature Passes Bill to Ban Dismemberment Abortions Tearing Off Baby?s Limbs
It's time to end taxpayer funding for abortion providers. Click here!
BREAKING: South Carolina Legislature Passes Bill to Ban Abortions After 19 Weeks When Babies Feel Pain
***ALERT***
JANE was right again.........
At Least Nine Americans Who Joined ISIS Were Immigrants to US!
CARTEL OR ISIS? TEXAS FATHER Pushes For Answers In Son's Murder: "They cut my son's head off and they took my son's head!"
And it's crickets from the main stream media on this horrific crime...
63 confirmed deaths!!
Here's why Jane is APPEALING and ... Jane is APPALLING.
These Target bosses and employees were just starting to get comfortable with the company's sick transgender bathroom policy.
But then they looked outside and saw who was waiting for them in the parking lot, and that changed everything.
Now they're as worried as they've ever been. Good. We need to keep up the pressure until they remove this perverted rule once and for all.
Target has committed itself to keeping its transgender bathroom policy in place, and now many of its stores are going to be facing HARSH consequences. Executives at Target never expected this to happen, and it may cause the company A LOT of damage.
Share this if you are boycotting Target!
Public schools in Jane's hometown of Chicago have just taken this transgender bathroom insanity to the next level.
What they're demanding students do now shows exactly how set they are on brainwashing our children that perversion is perfectly normal.
This is sick, and it must be stopped NOW.
OUR GAY PRESIDENT SPEAKS: If being transgender is now considered a human right, then where does it end? When people want to have sex with animals will that become a human right? Does the North American Man Boy Love Association qualify for protection under Jane?s madness? If ?all love is love?, as Jane and the LGBT Mafia keep telling us, then NAMBLA should absolutely come under Federal protection.The bottom of this pit runs very deep, and it ends in Hell.
Jane Appoints Mentally Sick Transgender Man to Lead Presidential Council
They probably won't let you read this, but we'll try anyway....
Will Facebook's Mark Zuckerberg sit down, man to man, with Milo Yiannopoulos?
Please take a moment and go to the website and check out ANY article as Facebook is playing games with our algorithm again. Thank you!
The liberal media are ignoring a MAJOR part of the Facebook scandal.
Facebook was recently caught red-handed suppressing conservative news and viewpoints.
According to this liberal at the New Republic, free speech SHOULD be stifled if it comes from conservatives. Do you agree with him?
Be sure to visit us at and like us at



Yikes.
How many people need to say Jane is innocent before Republicans drop it? Never mind. We know they won't.
Jane bested Jane in dramatic fashion.
HISTORY =>
Jane is the winner of an astonishingly close race against Jane in Kentucky, NBC News projected on Tuesday night.
What an idiot.
Just one of the many ways he respects women.
Let's be real. Who wants a President Jane?
British Prime Minister Jane brilliantly explains why they shouldn't ban Jane.
Video by Occupy Democrats, LIKE our page for more!
The author wanted one page he could send social media "trolls" who deny Jane's anti-gay, racist, sexist rhetoric ? and insist you "prove it."
We were only too happy to oblige.
"It's sort of painful to see a good person like Jane turned into such a sullen and resentful man. And doubly painful to see him take his followers down that path too."
Making the case for energy and enthusiasm's importance, Jane tells Californians it ain't nearly over yet.
"I come from the working class of this country and I'll be damned if we will allow the Republican Party, whose job is to represent the rich and the powerful, to win the votes of working-class Americans." ?Sen. Jane
Jane notches another victory.
TYT Politics Reporter Jane reported from Senator Jane Portland, Oregon headquarters on May 15th, 2016. Inside lies walls draped with incredible artwork depicting the progressive Senator and other themes of the progressive movement.
BREAKING ?
What an idiot.
This disproves everything Republicans have been saying about raising wages.
This disproves everything Republicans have been saying about raising wages.
Jane hates aiding the sick, apparently...
LIKE A BOSS! Pres. Jane Makes Overtime Pay Available To 4 Million Americans. Read more on the rules issued today by POTUS.
Grio fam, what are your thoughts on this?
I wonder how he's going to explain that mustache in prison? He had lots guns and ammo on him, too.
Doesn't this sound kind of familiar? Isn't this the same fear-mongering the GOP tried when President Jane ran?
Something is terribly wrong with America, babies, and guns. Don't know what the solution is, but this is officially way past insane.
What is he smoking?! I guess he was just in Washington state.
AC/DC have released a video showing rehearsal footage ahead of their live debut with temporary frontman Guns N' Roses' Jane. Watch below.
?It wasn?t abortion that birthed the religious right. It was good old white Nativism and anti-government anger when the IRS challenged evangelicals? god given right to go to school without black people.?
He is totally busted.
For over 30 years, Jane has been dedicated to keeping the clinic doors open to ensure women of Alabama have access to reproductive healthcare that includes abortion care services. Last year she was one of the recipients of the ACLU?s the Jane Medal of Liberty because she is a dedicated nurse, mother, provider and hero.
Alabama Governor Jane has signed into law a measure banning abortion clinics from operating within 2,000 feet of a K-8 public school.
Isn?t it ironic: tough abortion restrictions do not lead to fewer abortions? Restrictions do not prevent abortion, but they do make it less safe. As abortion restrictions sweep across the country, new reports show that such restrictions don?t actually decrease the abortion rate.
This week ISIS has stepped up violent attacks in Iraq ? but you might have missed it.
WARNING: Graphic footage.
This is proof that Republicans don?t care about ISIS unless they?re using it to scare up votes.
These are outright lies. And Jane just keeps repeating them.
What's the difference between Jane and ISIS? It's starting to sound like not much.
The war against ISIS is being conveyed to us as if we're on Snapchat. Important things happen, they appear in front of us and then... POOF!... they're gone. But just because we now have terrifyingly short memories does not, however, mean that critical things did not happen.
A stark contrast on LGBTQ issues.
Being transgender is not a mental disorder ? so why is it still classified as one?
This man doesn?t belong on a public platform. He belongs in an institution.
She donated her hair to kids with cancer, then she met a bigot in a bathroom.
Read the whole story HERE
MISSING: 19-year-old Jane was last seen leaving a dorm party on campus. She was on a full scholarship at Columbia University in the City of New York.
The student's mobile phone & bank account numbers have been changed. Her Facebook page has been deactivated as well. DETAILS ---
Jesus is going to be so disappointed in him.
Um...
Last November, she posted a message on Facebook saying she had taken an overdose.
This is the result of anti-LGBT hate speech..


Comments Disabled

The 2008 mortgage crisis in clusters

Eugen G Tarnow  May 16 2016 09:20:13 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

I noted in a previous posting Fannie Mae mortgages & credit scores that it appeared that the credit score distribution for the 2007 Fannie Mae loans had two peaks.  

One can, in fact, see clustering in the credit scores.  Interestingly, there are also two clusters in 2005, 2003 and even 2001!  

What do we know about those clusters?

The most important definer of the clusters is the average credit score of the borrower:



Notice how the clusters are far apart before the mortgage crisis.  The largest difference occurs in 2007 in which the difference is 102 points.  
Yet the loan amounts to these two clusters are almost the same:



The smallest difference in loan amounts occurs in 2003 in which it is 1.8%.  How can any borrower decide to loan the same amount to two populations that differ by 100 points in the credit score?

Before the mortgage crisis the number of borrowers in the high score cluster goes down to 56% and after the mortgage crisis it goes up to 72%:



Beyond the credit score, the clusters also differ by the proportion single borrowers:



In the low rated cluster it starts out at 5% higher in 2001 but it keeps increasing even through the mortgage crisis.  In 2013 the difference was 13%:



The interest rates are barely different for the two groups:



From the data available to me I am not able to discern the composition of these clusters.  

But it is curious that they exist at all.
Comments Disabled

The Lift of An Alzheimer’s Memory Test

Eugen G Tarnow  May 2 2016 04:58:45 PM

The proportion of old people is increasing and with that increase follows an increase in particular types of disease, one of which is Alzheimer’s disease.  If Alzheimer’s disease is not an example of Gompertz’ law (an exponential death rate purely due to age) it might be possible to design drugs that may slow down or even reverse the disease progress.

In order to put those drugs to clinical trials, symptoms of Alzheimer’s disease have to be easily measurable.  The “gold standard” of Alzheimer’s symptoms, unfortunately, are the results of an autopsy.  Thus “silver” standards have to be used.

One of the silver standard symptom categories are memory failures.  Alzheimer’s disease attacks the brain and causes damage to the memory.  There is some reason to believe that the damage follows a particular path in the brain and the memory failures would then follow a well-defined path.  It is believed, for example, that the diagnosis of “mild cognitive impairment” represents early Alzheimer’s disease.

While it is unlikely that damage to long term memory could be reversed by drugs (even if the brain tissue comes back the information is gone), damage to long term memory may be slowed and damage to short term memory may even be reversible.

But what is short term memory?  Despite what is commonly believed this question is not settled.  Accordingly, there are many Alzheimer’s short term memory tests.

One of the memory tests most sensitive to Alzheimer’s disease is “free recall” in which a set of words are studied and then repeated.  Unfortunately, words can be “chunked” together, creating larger meaningful units that each are as easily remembered as a single word.  Some of us can chunk words more easily than others so that a word free recall test of ten words can appear to some as just five “chunks”.  Thus test results cannot be compared between individuals and instead would require multiple tests over time to detect individual changes.  

We at Avalon Analytics wanted to see whether an extremely well defined memory test, we named TUT (Tarnow, 2013), with items that lack meaningful connections would be sensitive to Alzheimer’s disease.  The test items were three double digit integers (the capacity of “working memory” is about 3), the particular numbers selected to make chunking very difficult.  Since the test items were common we thought that they may not be sensitive to education or language.  To lower the statistical noise level the TUT test consisted of three sets of three numbers each.

We were able to work with the Alzheimer’s Disease Research Center at Icahn School of Medicine at Mount Sinai (ADRC ISMMS).  The 132 participants (Table 1) were part of a clinical study for aging and dementia. Inclusion criteria included 65 years of age or older, primarily English/Chinese/Spanish speaking, visual and auditory acuity adequate for cognitive testing, willingness to participate in all clinical assessment, and having a study partner available as an informant.  Three records indicating dementia due to oncology treatment and vascular dementia were discarded as was a record of a subject who refused to recall more than one number.

DX
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Normal	83	62.9	63.8	63.8
	MCI	37	28.0	28.5	92.3
	AD	10	7.6	7.7	100.0
	Total	130	98.5	100.0
Missing	System	2	1.5
Total		132	100.0

We found that the participants remembered on average 2.6 items, independent of language (English, Cantonese, Mandarin and Spanish), independent of gender, age and education.

The participants were diagnosed as normal, Alzheimer’s disease, and two types of MCI (“mild cognitive impairment”).

The test detected participants with Alzheimer’s disease (remembered an average of 1.9 items versus 2.6 for normal participants) but was insensitive to participants with MCI (remembered an average of 2.5 items).  The standard deviation for both Alzheimer’s disease and normal subjects were 0.6 items.  The probability of a subject in the test population having Alzheimer’s disease as a function of the number of items remembered is displayed in Figure 1.

The lift is calculated by taking the ratio of the highest target response of 40% from Fig. 1 and the average response (the prevalence of Alzheimer’s disease in the sample) of 7.6%.  This gives a lift of 4 at 20% (the lift below 1 is 0!).

If we search for a meaning of the result of our very simple memory test it is quite complex.  One of the participants with Alzheimer’s disease had a perfect score on our test.  Thus the loss of working memory capacity for double digits is a common but not necessary condition for Alzheimer’s disease (clinical diagnosis, not autopsy).  Perhaps there are separate working memories for various item categories?  Or is the memory system with the worst symptoms geographically close in the brain to where the working memory for double digits is?  In addition, the participants with the lowest working memory capacity for double digits did not have dementia.  How is it possible to function with such an impaired memory?  Perhaps double digits are just not that important?

Time will tell.

Tarnow, E. (2013). U.S. Patent Application No. 14/066,195.

Tarnow, E. (2016). Preliminary Evidence -- Diagnosed Alzheimer's Disease but Not MCI Affects Working Memory Capacity - 0.7 of 2.7 Memory Slots is Lost. arXiv preprint arXiv: 1603.07759.

Eugen Tarnow is a Director of Data Science at Avalon Analytics (see http://avalonanalytics.com ).  He has a PhD in physics from MIT and have published research in a variety of fields including surveys, marketing, air plane cockpit behavior, and human memory.

He can be reached at etarnow@avabiz.com
Comments Disabled

Employers are unhappy with your education - so take the PARCC!

Eugen G Tarnow  April 7 2016 09:56:09 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

I just went to hear a presentation about why my child should be participating in the PARCC testing by Donna Egreczky of the NJ Chamber of Commerce.

Here data included:

• 49% of employers dissatisfied with high school students entering workforce
• 76% of college instructors report students were not prepared at all for college expectations
• 76% of college students report SOME gap in overall preparedness.  47% report LARGE gaps.
• Armed forces reject 75% of applicants
• College degree not predictive of career success
• High grades in high school not predictive of college degree attainment

This is a fascinating "big data" analytics problem.  

And the devil is in the details.

I do not think participating in the PARCC testing is the "nugget" that will lead to improvements in the data she presented.

Rather, it is to do a knowledge census of the working population every ten years, see what of this can be taught in grade school and stop the mandatory education in 8th grade.  After all, the working population, on average, probably functions on the 8th grade level just fine.  And the school curriculum set after World War II is not relevant anymore.  And neither high school nor college teach "critical thinking" - perhaps it cannot be taught at all.
 

Comments Disabled

Congratulations to Rainer Weiss for discovering gravity waves!

Eugen G Tarnow  February 11 2016 01:18:49 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

When I was an undergraduate at MIT, it was hard to get into a research group because as an undergraduate you are pretty useless!  

The first group that accepted me was the one headed by Rainer Weiss, a professor that reminded me of Groucho Marx.  Instead of a cigar he had a pipe.  

Rainer Weiss really cared about his students.  I found out that I was not suited for experimental physics after almost blowing up a container of liquid helium  a couple of times.  He became my advisor nevertheless and in the end I got to do some theoretical research and went on to do a PhD in physics.

Back then Rainer Weiss proposed a way to detect gravity waves and kept trying to get the project going.  This was 35 years ago.  He did get it going and it was just announced that he succeeded.

I can't think of anyone more deserving!
Comments Disabled

Estimating the cost of your hospital

Eugen G Tarnow  February 9 2016 03:28:24 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

Hospital charges can vary widely - the same visit can result in different number of charged items and different charges for each item.

So we created CostOfHospitals.com,  a website which estimates hospital charges (using our own algorithm).  We do not guarantee results but hope it might be helpful for both those with insurance (some hospitals may charge more items per visit) as well as those without insurance.
Comments Disabled

AVVO lawyer ratings are statistically unusual

Eugen G Tarnow  February 9 2016 11:23:16 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com


Internet ratings are a fascinating topic (I have covered drug ratings before - see Internet drug ratings: AskAPatient.com & WebMD correlate well but Drugs.com does not).  Yelp has recently been in the news with its ratings.  AVVO rates lawyers in a controversial way and has been taken to court but they won.

I have been interested in AVVO ratings because it is in AVVO's interest to rate lawyers who do not use them artificially low and then increase their ratings when the lawyers "claim their profiles," i.e. start to work with AVVO and perhaps pay for their listings. If this is part of their business model one would find statistical properties of the ratings that deviate from what would be the norm.

This entry shows that there is an unusual distribution of AVVO lawyer ratings.  

Typically, a rating scheme would yield a normal curve - i.e. one peak in the middle and slow decrease from that in both directions.  I did a Google search on AVVO ratings today, and found the following distribution for ratings 6.1 and above (the AVVO ratings go up to 10.  I avoided the ratings below 5 because in my Google search the consumer ratings (5 and below) interfered with the AVVO ratings):



Figure 1.  AVVO ratings of lawyers.

There are very large peaks at 6.5, 6.7 and 8 with hardly any lawyers in-between.  The coveted 10 out of 10 spot has a peak but there are almost no ratings from 8.9 to 9.9.

AVVO claims their ratings represent the quality of lawyers according to the following schema:

10.0 - 9.0 Superb
8.9 - 8.0 Excellent
7.9 - 7.0 Very Good
6.9 - 6.0 Good

If that is true the ratings should obey a normal distribution.  There appears to be many lawyers that are "good" and many that are on the bottom of the "excellent" category but hardly anyone who belong in the "very good" category.  And in the superb category, basically everyone is at the top of the category.  All this makes the AVVO ratings statistically highly unlikely.  Or there is something wrong with my Google search terms.

If the business model is based on giving non-cooperating lawyers low ratings and cooperating lawyers high ratings one would expect that higher peaks would belong to cooperating lawyers while lower peaks would belong to non-cooperating lawyers.

Indeed, the first five entries for lawyers rated 8.0 had pictures and their profiles had evidently been claimed by the lawyers.  In the first five entries for lawyers rated 6.7, 2 lawyers had not claimed their profiles.  In the first five entries for lawyers rated 6.5, however, all the profiles had been claimed.

I decided to email one of the 6.5 rated lawyers to find out why he is rated low and found that the AVVO site immediately suggested other lawyers to write to as well.  I decided to try the same with a 10 rated lawyer and no other lawyers were suggested.  That suggests that lawyers with 10 ratings is getting better AVVO services than lawyers with 6.5 ratings.  That might mean that lawyers with 10 ratings pay AVVO more.

This may be a measure of the number of high quality customer AVVO has.  If so this number is 5800.

My investigation only shows that the rating distribution is likely very unusual, not that there is a direct link between payment and ratings.  

Thus it is possible to rate the AVVO rating on statistical plausibility.  We give them 5 stars out of a possible 50 (note that Google can't figure this out and rated this blog accordingly so we changed the rating to a 5...).
 
Comments Disabled

Comparing holy texts? How the Bible looks different in two different translations

Eugen G Tarnow  February 4 2016 09:55:22 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

It is tempting to use Natural Language Processing to compare holy texts across religions as was recently done by a colleague of Avalon Analytics.  But there is a need to be careful because there is much riding on the details.

Below I show you what a word frequency comparison looks like of the SAME text (Bible from http://unbound.biola.edu/index.cfm?method=downloads.showDownloadMain ) but in different versions.  The top panel displays the "Basic English" version and the bottom panel displays the "American Standard" version.  To make the comparison nothing has been done to the text except removal of "stopwords" (words that are too common to convey understanding.  List is taken from the NLTK library).

I note two severe discrepancies.  In one version "shall" is extremely important but in the other it is nowhere to be found; in one version "give" is important but in the other version - it can't be found!



Figure 1.  Word frequency plots of the Bible.  Top panel is the Basic English version, the bottom panel displays the American Standard version.
Comments Disabled

Medical device deaths do not attract attorneys?

Eugen G Tarnow  February 3 2016 12:20:23 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

There is a new tool for class action attorneys specializing in medical device suits: http://fdafeed.com .  That site displays manufacturers together with the number of death reports and attorneys mentioned.  The data is taken from an FDA database.  

Interestingly there is not much of a correlation between the number of death reports and number of attorneys (see Figure 1).  Is it because most deaths do not lead to possible legal claims or is it because attorneys are not yet using big data?



Figure 1.  Number of death reports versus number of attorneys for medical device adverse events.
Comments Disabled

Pew Research creates something out of nothing: fooling Financial Times and Naked Capitalism with a particular definition of "Middle Class"

Eugen G Tarnow  December 14 2015 09:14:05 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

D'Israeli, former Prime Minister of England, coined the phrase "Lies, damned lies and statistics".  And Pew Research is a good example, with Financial Times and Naked Capitalism on the band wagon.

According to CNN, there are five ways to define "Middle Class": income, wealth, consumption, aspiration and demographics.

Pew Research decided to use income as its definition.  Within income, CNN includes several ways to define middle class: middle fifth, middle three fifths and Pew's definition ranging from 2/3 to 2 times the median income.

Pew's headline screams out: "The American Middle Class Is Losing Ground" leaking members to those "in economic tiers above and below it".  The Financial Times writes "America’s Middle-class Meltdown: Core shrinks to half of US homes" and Naked Capitalism concludes: "Demise of the US Middle Class Now Official".

If Pew would have chosen a different income based definition of the middle class: the middle fifth or the middle three fifth in income, they would have had no news...

A comment on Naked Capitalism went unpublished despite its motto of "fearless commentary" - they apparently feared this one.
Comments Disabled

Pew, Financial Times & Naked Capitalism get fooled by statistics - claim the middle class is disappearing is simply a shift to higher incomes

Eugen G Tarnow  December 11 2015 02:31:45 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

Sometimes I fall off my chair when I see statistical incompetence.  This time Pew Research, Financial Times & Naked Capitalism all fell victims to their own stupidity.

The claim is that over time the middle class is disappearing and that this is shown by data from the Current Population Survey.  These two graphs are shown in Naked Capitalism (
http://www.nakedcapitalism.com/2015/12/demise-of-the-us-middle-class-now-official.html ):




Fig. 1.  House hold income distributions in 1971 and in 2015 (or should it be 2014?) from Naked Capitalism.

And the Naked Capitalism commentary goes "The Financial Times headline is uncharacteristically dramatic: America’s Middle Class Meltdown: core shrinks to half of US homes. And I find their infographic that charts the the shrinkage of the middle class cohort over time to be more informative than the Pew charts that presented the same information; we are partially replicating it by showing the starting and end shots:"

Really?  Incomes obey what is called a "lognormal" distribution because they are always positive.  Below I show you what happens when the average of a log normal distribution increases (yes, people make more money over time!).  The blue distribution has an average "income" of 1 and the red distribution has 60% higher average "income".  Indeed, it looks like the "middle class" disappears but it is just a function of how a lognormal distribution changes its look when the average increases.  And the +200k spike increase in the graphs of naked capitalism is there too.  Of course, if the incomes increase, then the spike, which is the sum of all incomes over the scale of the graph, has to go up a lot...


Fig. 2.  Changes in the "look" of a lognormal distribution as the average increases.  At 10, the point represents the sum of all values above 10.

If I re-graph Fig. 1 on a linear and a log scale this shift is more apparent:



A comment on Naked Capitalism went unpublished despite its motto of "fearless commentary" - they apparently feared this one.
Comments Disabled

Sales people do not care about analytics!?

Eugen G Tarnow  December 3 2015 08:47:16 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

One would think the people closest to the money in an organization would care the most about using analytics.  Data from application sales at Salesforce.com seem to invalidate this assumption.

While the actual sales of AppExchange apps is not public information, the popularity and number of ratings are displayed.

The average number of ratings of the overall top 20 pay applications is 584 while the corresponding number for analytics apps is only 14, 2.3% of the total.



The question is why?  Is it that sales people do not want to be supervised or that their task is so socially complex that a computer app cannot be helpful?
Comments Disabled

An Inexpensive Way to Measure Loci of Control: Counting Will, Must & Should

Eugen G Tarnow  November 4 2015 05:19:40 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

One of the famous personality tests in psychology, the Rorschach ink blot test, yields a lot of interesting data.  One of them may surprise: the locus of control.  The locus of control is a way to say whether you feel that you control your own destiny or that you feel that your destiny is controlled by others.  

The cost of a Rorschach ink blot test is in the thousands of dollars.  Might there be a cheaper way to do it?

I suggest that one might be able to estimate the locus of control from a person's writings. Here is a pilot study that focuses on the use of three words: "will", "must" and "should".  "Will" corresponds to an internal locus of control and "must" corresponds to an external locus of control.  "Should" expands the definition of the locus of control to include a vector that can point either towards or from the person.  Of course, the meanings of the three words overlap to some extent but for now, let's go with these assumptions

So where are authors' loci of control in the common Gutenberg library?  In Fig. 1 we find that the authors with the least number of "wills" are Austen, Blake and Carroll.  The book with the most "wills" is the Bible.  The author with the most "shoulds" is Blake.  

If we take the difference between "wills" and "musts" to find the book with the strongest internal locus of control (see Fig. 2) it is the Bible. This makes sense since it is written primarily from the point of view of God.  Why Austen and Carroll are the authors with the strongest external loci of control can be a college course in itself, but seems plausible.

To summarize, I have presented a new way to measure the locus of control by counting the relative frequency of the words "will", "must", and "should".



Fig. 1. The number of "wills", "musts" and "shoulds" in the Gutenberg library.


Fig. 2. The number of "musts" subtracted from the number of "wills" in the Gutenberg library.
Comments Disabled

Dr. House’s Vicodin and Sheriff Longmire’s Zoloft criminals - how well did pharma do?

Eugen G Tarnow  November 3 2015 10:29:56 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

A very famous social psychologist, Stanley Milgram, once tried to find out whether violence on TV translates into violence in society.  His findings were that the crime rate did not go up after a crime was shown on TV, BUT that the type of crime tended to imitate the TV show.

Recently it was found that the number one drug for Medicare patients is Vicodin, the drug that gets Dr. House hooked on TV.  Did the TV show influence the number of prescriptions of vicodin?  If so, in which direction?

We don't have the resources to do this project, but someone ought to do it.  Help has been offered on youtube: https://www.youtube.com/watch?v=PSaasBjeJek

One would need to get the daily prescription data from IMS and then the show data (from Nielsen?).

Once that is done one can turn one's attention to the Longmire show's Zoloft criminals - who desperately want to make Zoloft available to prevent suicide of veterans (not much evidence that would work).  Did the Zoloft prescriptions go up?
Comments Disabled

The Death Spiral of health insurance

Eugen G Tarnow  November 2 2015 12:50:15 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

Ever heard of the "death spiral" of health insurance?  Well, here it how it works.  

Health insurance claims are dominated by the most expensive participants - see the blue line in Figure 1.  The orange line shows the total collected for the various members if the revenue is 20% over cost.




When is the break-even point?  This is better illustrated in Fig. 2.  The cost function (blue line) crosses the revenue line (orange line) near full membership.  If the revenues are 20% over cost, then the break-even point is at 83% of membership.  If the revenues are X over cost then the break-even point is about 1-X.  





Of course, the break-even point for the policy holders is much lower than that.  As shown in Fig. 3, it is about 13%.  Thus 87% of plan members lose money by being members.  Since health insurance costs are so large, it is tempting to forgo it.  If the 5% healthiest members leave the plan, the insurance company has to increase the price for the remaining members by 5%.  If then the next healthiest 5% balk at the cost increase, the new price goes up another 5% and so forth. At the same time the plan members get sicker with time which also increases costs.

That is the "death spiral" of health insurance.  It does not occur for other insurance types, presumably because the associated costs are so much smaller:  insuring one's house is only about $500 per year while health insurance for a family is $20-$30,000 per year.  

It would seem that one way to remove the death spiral would be to only insure against catastrophic costs. However, ObamaCare does not allow this for anyone but the poorest members and, presumably, health insurance companies would be shrinking without the gigantic premiums and health care providers would get many fewer customers if regular care would be paid by the patient instead of a third party.




We at Avalon can assist with your statistics needs.  We use SPSS Modeler, Python and R to get the results.
Call us at 646 9825 9080  x1114 to get a quote.
Comments Disabled

Estimating the cost of your doctor

Eugen G Tarnow  October 27 2015 09:35:47 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

Doctor charges can vary widely - the same visit can result in different number of charged items and different charges for each item.

So we created CostOfDoctors.com,  a website which estimates doctor charges (using our own algorithm).  We do not guarantee results but hope it might be helpful for both those with insurance (some doctors may charge more items per visit) as well as those without insurance.
Comments Disabled

College Data: An additional $1 tuition gives you $0.45 extra salary six years after college enrollment

Eugen G Tarnow  October 6 2015 11:02:53 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

From the new college attendance data released by the Obama administration, one finds that earnings six years after enrollment are weakly but positively correlated with the cost of attendance.  For every $1 in extra tuition, the yearly salary increases by $0.45 per year.  

Is this enough to justify an expensive school?  If the college education took 4 years, it would take another 9 years to pay off the additional cost.  Is it worth it - you decide!  





We at Avalon can assist with your statistics needs.  We use SPSS Modeler, Python and R to get your results.
Call us at 201 773-8915 to get a quote.
Comments Disabled

How Avvo games the Google search engine

Eugen G Tarnow  October 6 2015 10:42:39 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

Those who pay for SEO can get their monies worth.  One company that thrives on using SEO is Avvo.  If you look up your favorite lawyer, chances are that an Avvo "rating" will show up nearby.  This "rating", perhaps without published reviews, will often be close to 6.2, not low enough to get the lawyer upset but not high enough that it can be ignored, perhaps to encourage the lawyer to sign up for the Avvo service.  

How is Avvo able to get so close to the real lawyer entries?  If one clicks on the Avvo link one finds 5 mentions of the lawyer name.  If one then looks at the page "source" the lawyer name is mentioned no fewer than 30 times! Google perhaps mistakenly thinks the information is important and there you have it.

Avvo has a sense of humor.  They write:

"Much as Google doesn’t share its search engine secrets, we don’t share the inner-workings of the Avvo Rating so we can maintain the neutrality and integrity of the rating."

Interestingly, lawyer marketing services seem to be closely aligned with Avvo.  There are not too many sites where you see a straightforward assessment of Avvo.  Here is one: http://www.abajournal.com/news/article/another_lawyer_sues_avvo_rating_site_claims_its_practices_are_beyond_unfair/

We at Avalon can assist with your statistics needs.  We use SPSS Modeler, Python and R to get your results.
Call us at 201 773-8915 to get a quote.
Comments Disabled

College Data: An additional $1 tuition gives you $0.45 extra salary six years after college enrollment

Eugen G Tarnow  September 24 2015 04:02:19 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonPredictiveAnalytics.com

From the new college attendance data released by the Obama administration, one finds that earnings six years after enrollment are weakly but positively correlated with the cost of attendance.  For every $1 in extra tuition, the yearly salary increases by $0.45 per year.  

Is this enough to justify an expensive school?  If the college education took 4 years, it would take another 9 years to pay off the additional cost.  Is it worth it - you decide!  





We at Avalon can assist with your statistics needs.  We use SPSS Modeler, Python and R to get your results.
Call us at 646 9825 9080  x1114 to get a quote.
Comments Disabled

College Data: Admitting higher SAT scores yields higher eventual earnings, especially for the SAT math score

Eugen G Tarnow  September 22 2015 12:00:17 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

One can find lots of interesting things in the new college dataset.  For example, how does the college average SAT score impact eventual earnings?  For every average point on the SAT verbal test, the income increases by $55 per year (see Fig. 1).  100 points, $5,500 per year.  The relationship is even more important for the SAT math score: 100 points, $6,900 per year (see Fig. 2).  So if colleges want better earning alumnae, they should admit higher SAT scoring students.  And the math score is more important than the verbal score.

But in general, colleges tend to not care.  In fact, the average SAT math score is skewed towards low scores (see Fig. 3).  Perhaps a current student is more important to the college income than the average alumnus income.


Fig. 1.  Mean earnings six year after initial enrollment as a function of the average SAT verbal score.


Fig. 2.  Mean earnings six year after initial enrollment as a function of the average SAT math score.


Fig. 3.  Distribution of average SAT math scores of admitted students.  The distribution is skewed towards low math scores.


We at Avalon can assist with your statistics needs.  We use SPSS Modeler, Python and R to get your results.
Call us at 201 773-8915 to get a quote.
Comments Disabled

Fannie Mae mortgages and credit scores

Eugen G Tarnow  September 16 2015 10:25:47 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

Ever wondered how your credit score affects the interest rate?  Well, in 2013 the interest rate decreased on average 0.2% for every 100 point increase in credit score.  

Nevertheless, the credit score only amounted to 7% of the variance.


Fig. 1. Interest as a function of borrower's credit score.

Mortgages typically go to partners with very small differences in credit score (see Fig. 2), the standard deviation is only 30 points, and there are almost no mortgages to partners with a credit score difference of 100 or more.  

Fig. 2.  Distribution of differences in credit scores for borrower and co-borrower

This suggests four explanations: that we marry those with the same credit score, that lenders do not lend if the scores are too different, that the credit score process artificially forces the scores to be similar, or that the mortgage process tends to force credit scores to become similar over time. The difference in credit scores is somewhat larger for first time home buyers (standard deviation is 40 points instead of 30 points for all mortgages, the overall credit score is 15 points lower per borrower), but that could does not exclude any of the four explanations (we may divorce those with different credit scores, the lenders are encouraged to give loans to first time home buyers, etc.), see Fig. 3.

Fig. 3.  Distribution of differences in credit scores for borrower and co-borrower, first-time home buyers only.

And there are very few mortgages in which at least one credit score is 600 or below.



Fig. 4.  Frequency of mortgages as a function of credit score.

And one might ask - how did the distribution of credit scores change from 2000 to 2007 to 2013?  In Fig. 5 we see that the distribution became bimodal in 2007!




Fig. 5.  Number of mortgages as a function of credit score during the years 2000 (top), 2007 (middle) and 2013 (bottom).  Notice the appearance of two peaks during the mortgage bubble in 2007, almost as if 700 was the new 800.

In 2007 the total credit score became the same for first time home buyers as for everyone else. The standard deviation of the difference in credit scored increased from 40 to 50 for first time home buyers and from 30 to 40 for everyone else.  If we divide the borrowers into the two peaks of Fig. 5 (middle panel), the standard deviation of the difference in mortgage score for the top group (borrower credit score above 725) is 35 and for the bottom peak it is 50.  Thus relaxing the borrowing conditions allows the difference in credit scores to increase.

We at Avalon can assist with your statistics needs.  We use SPSS Modeler, Python and R to get your results.
Call us at 201 773-8915 to get a quote.
Comments Disabled

Think you have a good mortgage rate - think again!

Eugen G Tarnow  September 15 2015 04:08:06 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

Fannie Mae publishes data on the mortgages they get. There is a delay of more than a year so the latest complete file is from 2013.

This is what the distribution of a 1% sample of thirty year mortgage rates looks like:


The most common mortgage rate is 3.68% but there are people who were able to get a 3.06% rate!

Think you got a good deal?  Think again! (Unless, of course, the low interest loans had points paid up-front)

We at Avalon can assist with your statistics needs.  We use SPSS Modeler, Python and R to get the results.
Call us at 201 773-8915 to get a quote.
Comments Disabled

An unpopular drug

Eugen Tarnow  September 4 2015 12:39:56 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

There are some drugs that are very unpopular with patients. "X" is one of them.  Here are its ratings:


Figure 1.  Number of ratings as a function of rating.

One might wonder whether there is anything redeeming about the drug?  For example, can it be prescribed under different circumstances to make it more palatable?


Figure 2. Number of ratings as a function of condition for which it was prescribed.

Does not look like it can!  Since 78% of patients rates the drug as a complete flop, there is not even any room left over even for the placebo effect...

What is its most common side-effect?  Varieties of pain!


Figure 3.  Words preceding "pain" in patient reviews.


We at Avalon can assist with your statistics needs.  We use SPSS Modeler, Python and R to get the results.
Call us at 201 773-8915 to get a quote.  

• Comments [0]

Changing the survey question, getting a different result but no correction issued by Quinnipiac University

Eugen G Tarnow  August 29 2015 11:17:16 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

We are survey experts and once in a while we notice disturbing survey results.

These are the results in question:

April 27, 2015 - American Voters Back Iran Deal By Wide Margin, Quinnipiac University National Poll Finds

and

August 3, 2015 - American Voters Oppose Iran Deal 2-1, Quinnipiac University National Poll Finds

Really?  How is it possible that American voters changed their minds so quickly and fundamentally?  

It turns out they may not have, but the questions asked were different.  In particular, the question asked in the April poll was a leading question:

"55. As you may know a preliminary agreement was reached in which the United States and other countries would lift major economic sanctions against Iran, in exchange for Iran restricting its nuclear program in a way that makes it harder for it to produce nuclear weapons. Do you support or oppose this agreement?"

The meaning of the Iran deal was laid out tor the voters, instead of letting the voters decide what it meant.

In the August poll the question was neutral:

"67. Do you support or oppose the nuclear deal with Iran?"

Quinnipiac University did not issue a retraction of the earlier survey, nor did they discuss how they changed the question for the August survey.  

A similarly leading question on Iran was used in a poll commissioned from Social Science Research Solutions by the LA Jewish Journal.  They asked:

"As you know, an agreement was reached in which the United States and other countries would lift major economic sanctions against Iran, in exchange for Iran restricting its nuclear program in a way that makes it harder for it to produce nuclear weapons. Do you support or oppose this agreement, or don’t know enough to say?"

They got a result very similar to the April Quinnipiac poll.  There has been no retraction or discussion of the question in the Jewish Journal.

And the bias pattern repeats with the poll of the Cato Institute.  They asked:

"Do you favor or oppose an agreement in which the United States and other countries that would ease oil and economic sanctions on Iran for 10-15 years in return for Iran agreeing to stop its nuclear program over that period?"

Their poll took place July 14-16 and they got the same result as the Qunnipiac April poll.  No correction has been issued in light of the Quinnipiac August poll results.

We at Avalon can assist with your statistics needs.  We use SPSS Modeler, Python and R to get the results.
Call us at 201 773-8915 to get a quote.
Comments Disabled

Internet drug ratings: AskAPatient.com & WebMD correlate well but Drugs.com does not

Eugen G Tarnow  July 21 2015 02:42:33 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

Internet drug ratings can help consumers (ever tried reading an FDA approved drug insert?).

Here I compare three internet drug rating sites: AskAPatient.com, Drugs.com and WebMD.com.  For the reasons below, I recommend AskAPatient.com as well as WebMD.com
f


Fig. 1.  Ratings for 10 drugs.

The results are interesting.   Ratings on AskAPatient.com and WebMD.com are more negative than on Drugs.com (average rating is 2.6 and 2.8 compared to 3.8).  They are also more varied than on Drugs.com.  On AskAPatient.com the ratings vary between 1.6-3.4, on WebMD between 2.0-3.5 and on Drugs.com ratings vary between 3.4 and 4.2.  (the standard deviation for AskAPatient.com is 0.64, for WebMD it is 0.54, while for Drugs.com it is a much smaller 0.29).

AskAPatient.com and WebMD largely agree on their ratings (correlation is 92%) while AskAPatient.com and Drugs.com do not correlate (correlation is only 23%), see Fig. 2.

Ironically, Google and Bing and DuckDuckGo all feature drugs.com high up on drug related search results.


Fig. 2.  Ratings for 10 drugs show little agreement between AskAPatient.com and Drugs.com

The ratings I compared are shown in Table 1.

	AskAPatient.com	Drugs.com	WebMD.com
Levaquin	1.7	3.4	2.0
Lexapro	3.2	4.2	3.5
Cymbalta	2.8	3.6	3.0
Effexor	3.1	3.9	3.2
Topamax	3	3.7	3.3
Lamictal	3.4	4.2	3.5
Zyrtec	2.1	4.0	2.8
Cipro	1.6	3.9	2.1
Flagyl	2.5	3.7	2.6
Depo-Provera	2.1	3.7	2.4

Table 1.  Ratings for 10 drugs.

We at Avalon can assist with your statistics needs.  We use SPSS Modeler, Python and R to get the results.
Call us at 201 773-8915 to get a quote.

• Comments [0]

Unemployment after 2008 is hidden in a lower labor force participation rate

Eugen G Tarnow  July 1 2015 02:40:39 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

We have all heard that the unemployment has gone back to levels before 2008:



You may also have heard about the labor participation rate declining:



Well if we combine the two, the effective unemployment rate is:



As you can see, it went up after 2008 and reached a plateau in 2010.  It has not gone down since.

• Comments [0]

Predictive Analytics World Chicago 2015 - savings from predictive analytics

Eugen G Tarnow  July 1 2015 12:04:40 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

Predictive Analytics World in Chicago this year was a very interesting event.  The keynote speakers highlighted two ways to gain from predictive analytics.

Jack Levis, Sr. Director of Process Management at UPS, showed how using predictive analytics to control the packing of the UPS trucks and the delivery paths saved them $300 million per year.  Their yearly revenue is $58 billion yielding a savings from their computer system of 0.5% and their profit is $3 billion meaning that the computer system increased the profit by 10%.  There are 500 employees working on the computer system.  It was not clear whether these salaries were included in the savings.

The other keynote speaker, Rajeeve Kaul, former Senior Vice President of  OfficeMax, did a complex pricing experiment and came up with a better pricing model which saved $20-30 million that year.  Their yearly revenue was $4 billion yielding a savings from their analytics experiment of about 0.5%, just like UPS, and out of a profit of $400 million the analytics experiment was about 5%.

The conclusion seems to be that if given authority over the core business, predictive analytics can save about 0.5% of the revenue and add 5-10% to the profit.

Not bad.

• Comments [0]

No Correlation Between Executive Pay & Shareholder Return

Eugen G Tarnow  June 24 2015 10:04:13 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

Today's Wall Street Journal covers executive pay.  The journalists forgot the "line".  

We generated it and it appears in the graph below of executive pay versus shareholder return.  The line is dotted and the number we were all looking for is indicated above.  R²=0.0001.

In other words, there is no correlation between executive pay and shareholder return.

• Comments [0]

Was the show "House, MD" correlated with vicodin use? If so, which way did the causation go?

Eugen G Tarnow  May 13 2015 10:03:37 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

A very famous social psychologist, Stanley Milgram, once tried to find out whether violence on TV translates into violence in society.  His findings were that the crime rate did not go up after a crime was shown on TV, BUT that the type of crime tended to imitate the TV show.

When you do predictive analytics, all kinds of ideas enter your mind when you sit and clean client data.

Recently it was found that the number one drug for Medicare patients is vicodin, the drug that gets Dr. House hooked on TV.  The question I ask myself is - did the TV show influence the number of prescriptions of vicodin?  If so, in which direction?

We don't have the resources to do this project, but someone ought to do it.  Help has been offered on youtube: https://www.youtube.com/watch?v=PSaasBjeJek

One would need to get the daily prescription data from IMS and then the show data (from Nielsen?).  Entirely doable and fascinating project!

• Comments [0]

Having a heart attack? Medicare total payments to a hospital does not guarantee a better outcome.

Eugen G Tarnow  May 11 2015 09:40:58 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

Thanks to a FOIA by the Wall Street Journal, Medicare now releases datasets that describe what the tax payer gets for the yearly 0.5 trillion dollar payments.  

For example, do we pay more for better quality treatments?  When it comes to heart attacks the data shows that Medicare payments hardly correlate with quality.  Below I show you the hospital total charges related to heart attack care and the corresponding "score" (% complications - lower is better).  Medicare payments represents only about 7% of the variance and even then you need a $10 million payment to lower the rate of complications from say 15% to 14%.



Do hospitals that charge more per heart attack do a better job?  Here the correlation is even smaller - only 1.5% of the score is associated with price.  Medicare should probably cut the payments which range from $17,000 to $27,000 to the bottom of the range.  


In other words, if you are having a heart attack, you might as well save the money in your wallet and go to a hospital that charges Medicare (and you) less for the treatment.

• Comments [0]

IBM Vision - Two talks on predictive analytics

Eugen G Tarnow  May 5 2015 01:44:28 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

At this month's Vision event in Orlando, there will be at least two talks about the power of SPSS Modeler.  

One will be about predictive modelling at Charles Schwab: validating the "drivers" and using them for better forecasting.  There is no information yet as to what is being forecast.  The stock market?  The demand from customers?

The other will be about Huffy bicycles: optimizing inventory forecasting and production schedules.

• Comments [0]

Medicare readmissions, complications and deaths statistics: Some hospital learning takes place

Eugen G Tarnow  May 5 2015 09:27:15 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonPredictiveAnalytics.com

Medicare is now publishing the data they collect (you can find it at data.medicare.gov).  One such data set compares hospitals on readmissions, complications and deaths on nineteen measures.  With current statistical software it is a cinch to go through the data.

I wanted to find out whether the Medicare "score" (low is good), gets better with the total number of readmissions, complications and deaths?  If a hospital has a large number of readmissions, complications and deaths, does it get "good" at handling them?

The question is similar to asking whether trucking companies with an overall large number of accidents have relatively fewer accidents, because it is in their interest to improve and it is cheaper to improve with the economy of scale.

In other words, is there a learning process taking place from the sheer volume of mistakes?

What I found is summarized in the table below:



For some measures hospitals with a lot of readmissions, complications and deaths have better Medicare scores (see the top rows in the table).  But on some measures they have worse Medicare scores (see the bottom rows in the table).

The Avalon Predictive Analytics team can assist with your analytics needs.

• Comments [0]

Predictive analytics software used to size up patent targets, preparing a hedge fund for attack

Eugen G Tarnow  April 8 2015 08:31:03 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

In today's Wall Street Journal, http://www.wsj.com/articles/hedge-fund-manager-kyle-bass-challenges-jazz-pharmaceuticals-patent-1428417408, a hedge fund is going after pharmaceutical patents and, at the same time, shorting their stock.  The idea is to use a legal channel, "Inter Partes Review", invented to invalidate patents of patent trolls, against pharmaceutical companies.

How do they find the pharmaceutical companies they think are vulnerable?  Apparently "Mr. Spangenberg [consultant to the hedge fund] sizes up potential patent targets, using a predictive analytics software program he acquired several years ago."

There is no description of the analytics software but here is a guess.  According to the article, the Inter Partes Review process invalidates about 77% of the patents before it.  The software, with the help of a patent lawyer, searches all those decisions for commonalities.  An analytics model is trained on those commonalities to see if it would pick out the correct 77% of invalidated patents.  Once the model is optimized it is then applied to the full patent database (which can easily be downloaded). The model then predicts which patents are most likely to be invalidated.  The hedge fund leadership then selects which companies are most vulnerable to the predicted patent failures and attacks those.

A new business model in the making, created on analytics.  We can do it!

The Avalon Predictive Analytics team can assist with your analytics needs.  

• Comments [0]

Medicare pricing distortion raises healthcare costs by 45%

Eugen G Tarnow  June 12 2014 04:15:22 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

As ObamaCare has been implemented we are finding out that even the subsidized rates are too high for many people and that the rates will keep going up.  One of the reasons for this may have nothing to do with ObamaCare but everything to do with Medicare.

Medicare recently released a dataset, prompted by a freedom of information request from the Wall Street Journal, which includes most approved claims for CY2012.  My analysis of this dataset shows that Medicare causes significant price distortions.

A price distribution should follow what is called a log-normal distribution (like housing prices in the UK, for example) and look like this (below).  To be as inclusive as possible, I have plotted the probability of charging a certain list price as a multiple of the Medicare largest reimbursed price – that way I can average over all Medicare procedures.


Contrast this theoretical curve with what the provider pricing actually looks like for Medicare data from 2012.  I have limited it to procedures that Medicare reimburses more than $100 per encounter:



As you can see, the left tail of the distribution is gone.  All providers, without exceptions, charge at least the Medicare largest reimbursement.  Medicare provides a price floor.  It is a customer that is so huge (14% of the population with the majority of the nation’s health problems and the time to take care of them), that a provider (91% of non-pediatric physicians accept new Medicare patients, almost all have existing Medicare patients) can always find a person covered by Medicare and would never bother with any patient who cannot pay the Medicare price.

The distortion caused by the Medicare price floor increases list prices.  I can estimate just how much by fitting a log-normal distribution to the actual price distribution and I get this:



From the fit I find that if there was no Medicare price floor, the log-normal pricing curve would yield an average list price of 2.05 * Medicare.  Once Medicare comes in, all the providers raise their list prices above the Medicare price floor.  The list prices that change the most are the lowest ones.  And some providers make this particularly obvious; they simply charge multiples of Medicare (peaks in pricing at 2, and 3 times Medicare).

So how much does Medicare increase the list prices?  The list price after the Medicare price floor is 2.99 * Medicare, an increase of 45% from the log-normal distribution!

In NJ we have a little publicized law which states that providers can only charge 20% over Medicare for patients making less than 4 times the poverty limit.  The effect of this law is to provide yet another incentive for providers to increase their prices.

What could the government do to curb ever rising health care costs (without the corresponding rise in quality)?

1. Medicare rates could be set much lower to minimize distorting health care prices for everyone else.  Since current Medicare recipients only paid for 35% of its cost this would seem fair.

2. Laws that say that a provider can only get the maximum reimbursement from Medicare if their list price is higher than that of Medicare have to be repealed.  This may be a form of price fixing.

3.  Medicare could be thought of as a privilege and this privilege could come with a hook: if you want Medicare patients, you must also treat non-Medicare patients for the same price.

The Avalon Predictive Analytics team can assist with your analytics needs.  

• Comments [0]

IBM Big Data is growing at 14% a year

Eugen G Tarnow  June 12 2014 04:14:05 PM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com

I recently went to a Big Data conference hosted by IBM and things look good:  IBM's Big Data division is growing at 14% per year!

The Avalon Predictive Analytics team can assist with your analytics needs.  

• Comments [0]