Exams and handwriting30mar99: Roderic Vassie: A slow writer wins through.
In the summer of 1976, when Roderic was 15, he took his 'O' level examinations with good results (including 'A' grades in French and Maths). But his result for English Language, was a 'U' (a bad fail). English Language is probably the most important 'O' level. It is a minimum qualification for many jobs and higher education.
Roderic's school, Archbishop Holgate's Grammar School in York realised that he had a particular problem with writing and arranged for him to be seen by an educational psychologist employed by the City of York.
As the attached note shows, Roderic was tested by the educational psychologist and shown to be intelligent with a good ability in English. But that Roderic was left-handed and had 'developed a most clumsy style of holding his pen' which made it 'impossible for him to write quickly'. He was 'not able to demonstrate this ability when in a situation that requires quick writing'.
In the re-sits in the following Autumn, Roderic was allowed 10 minutes extra in every hour for his exam and a note was added to his exam scripts that mentioned his problem. His grade for English Language improved from 'U' to 'C' (a reasonably good grade). He was later allowed extra time in 'A' level and BA examinations. He went on to postgraduate study and has a PhD.
Recently Roderic agreed to try a small test for faxfn. Trying to write as fast as possible he wrote the following well-known rhyme:
Mary had a little lambOn four trials he took 39, 35, 37 and 40 seconds. Usually people are more consistent, testing slower on the first (practice) try and having a constant speed subsequently. Usually the practice time is discarded. Averaging his non-practice times gives 37secs.
faxfn has informally tested other people from time to time (probably about 100) using this test and would classify the times as follows:
fast 20 - 24 secRoderic has been told that correct handwriting teaching at an early age could have made his problem easier. He is left-handed and was initially taught with a pencil to do italic writing and with the change to a fountain pen, he found pushing the pen into the paper caused some difficulty. In addition, his hand following over the newly written ink created smudges. There are better techniques that can be used for left-handers improve their writing. But these are much harder to teach in later years.
Other workarounds for problems with handwriting include such famous examples as Leonardo da Vinci's mirror writing. Perhaps more research in the area should be undertaken and schools and parents should be more aware of the problem and possible solutions. While not perfect the 'Mary Lamb' test can be used as a quick test to spot potential examination difficulties.
03apr99: Computer Consultant: A slower writer wins too (eventually)
This computer consultant kindly agreed to do the Mary Lamb test (see above) for faxfn. He obtained the following times: 43.6 sec, 44.4 sec and 44 sec. According to the classification given above this puts him in the extremely slow category.
In the summer of 1976 he passed his Physics O-level (grade C) but failed English Language with the worst classification ('U' for unclassified). In 1976 he passed two further O-levels (Maths and Technical Drawing) but failed English Language and English Literature. He was awarded a CSE grade 3 in English Language.
On leaving school, he became a trainee aircraft technician with the RAF. During six years service he 'learnt a bit' including some data analysis. Following his time in the RAF, he attended technical college and obtained an ONC (with distinction) in Computer Studies. Results were awarded mostly by course work so writing speed was no problem.
This was followed by 4 years programming (COBAL, Basic, C etc) followed by a one year HNC course where in most of the topics he got merits. After another five years of writing software (and becoming a company director) he returned to education to do an MSc in Information Processing. This was not so successful: he scraped a pass. But the course did have several time limited exams that were the main means of assessment.
Since then he has worked for 4 years as a consultant/developer and clearly is a man of significant skill and talent. But still has no O-level in English.
His final comment was "I have been extremely lucky to have had the opportunity to carry on with an education. Careers guidance at school had me stitched up as a tyre fitter."
11oct99a: Find the reference: Handwriting style affects exam results.
Faxfn has been told of a paper published in the 1960's or 70's, possibly from someone at the Open University about writing style and examination performance.
The research took ten different answers to a hypothetical examination and wrote each of them in ten different handwriting styles. A panel of teachers were asked to mark the exam answers. They all had the one set of answers (i.e. copies of the original ten) but written in a different combinations of handwriting style. None of the panel knew the purpose of the project.
Analysis of the marks given showed that it was a significant advantage to have a certain style of writing that looked "adult". Surprisingly writing that was too neat was penalised as was (less surprisingly) scruffy writing.
Faxfn would like a reference to this work.
11oct99b: Faxfn Test Dept: Fast writers, with first class degrees.
Recent testing of Mary Lamb speeds. 19 seconds is the fastest ever tested.
* Geoff Kendall. He has indistinct but intelligent looking writing. He misses out many of the squiggles forming individual letters. But in examinations he printed and underlined key words, presumably to give the maximum effect for speed reading examiners.
23oct99a: Faxfn Test Dept: Three more tests. (the control?)
Having tested three people with first class degrees, faxfn staff did a random walk to a local shop. We found 3 young women: a schoolgirl doing A-levels, her aunt who is an undergraduate and their friend who is a deputy manager in a medium sized branch of a national chain of shops.
The schoolgirl had slow writing (a ML time of 30 sec), the undergraduate had standard speed writing (ML 27) and the deputy manager had very slow writing (ML 38). The deputy manager also thought that "exam panic" had been part of her problem.
These three writing speeds of "random walk" selected people (ML 27, 30, 38) are slower than the writing speeds of the three people selected for their first class degrees (ML 19, 19 and 25).
For the record the schoolgirl had 10 GCSEs grades ranging from B to E (the lowest grade of pass was F); the undergraduate had five A-levels grades ranging from B to D and the deputy manager's exam results had been poor (a few low grade GCSEs and O-levels). After school the deputy manager briefly trained as a nurse but dropped out. However, when the national chain was recruiting, she got past the initial rejection by her enthusiasm and persistence. Starting as a Retail Sales 1 she was promoted up several levels to Deputy Manager (Sales) in just over a year.
Exams, Competencies and Psychometrics19aug99: Recommended Reading: "IQ did not contribute to GCSE performance"
This headline from a well-known broadsheet exaggerates the findings of the research reported in "Memory, IQ and Exam Performance" (by John Wilding, Elizabeth Valentine, Peter Marshall and Susan Cook. Educational Psychology, Vol 19, No 2, 1999). But a fuller quote still shows an interesting story:
"In combination, therefore, the two studies support the existence of a general memory ability, independent of IQ, and indicate that this ability accounts for between 10 and 20% of the variance in exam performance. Surprisingly Cattel IQ did not contribute to GCSE performance in either group. There was of course a large difference in both GCSE performance and Cattel IQ between the groups, but the multiple differences between them make it unwise to ascribe the differences in GCSE performance solely to Cattel IQ."(More later)
22aug99: Competencies or MBAs?: "You can teach a turkey to climb a tree, but it's easier to hire a squirrel."
An interesting booklet published by Hay/McBer Research Press in 1994 has come the notice of faxfn. "Competency Assessment Methods" by Lyle Spencer, David McClelland and Singe Spencer. Quotes:
"... an increasing number of studies were published which showed that traditional aptitude and knowledge content tests, as well as school grades and credentials:and
"While changing motives and traits is possible (McClelland and Winter, 1971), the process is lengthy, expensive and difficult. From a cost-effectiveness standpoint, the rule is 'hire for core motivation and trait characteristics, and develop knowledge and skills.' Most organisations do the reverse: they hire on the basis of educational credentials (MBAs from good schools) and assume the candidates can be indoctrinated with the appropriate motives and traits... in the words of one personnel manager, 'You can teach a turkey to climb a tree, but it's easier to hire a squirrel.'"You may like to order your copy from Hay/McBurr Research Press, 0207 730 0833.
24jul99a: Academic A: They learn then forget
(This academic teaches business studies to undergraduates and also teaches on an amber accredited MBA course.)
... [Undergraduates] may have passed exams but what are we testing? It's just regurgitation. They learn it six weeks before the exam and forget it afterwards. And they will end up doing jobs for which the course has no relevance...
(See in full Undergraduates lack initiative)
05sep99: Elizabeth Valentine: "Memory, IQ and Exam Performance" - a summary
MEMORY, IQ AND EXAMINATION PERFORMANCE
John Wilding and Elizabeth Valentine
Royal Holloway, University of London
Educational Psychology, vol.19, no.2, 1999, pp.117-132.
Relations between memory ability, intelligence (IQ) and GCSE performance were examined in two groups of 16-17 year-olds of above average intelligence, one consisting of members of Mensa. Memory (immediate and delayed, i.e. after a week) was tested on a wide range of materials (including faces, a story, a complex visual figure, words and numbers) and by self-rating. Intelligence was measured using the Cattell Culture Fair Test.
Evidence was obtained for a general memory factor independent of intelligence, in that rank orderings of performance were similar across the range of memory tasks and correlated with self-reported memory ability, with differences in IQ removed. Measures of memory ability were significantly related to GCSE performance whereas IQ was not. This suggests that GCSE require the reproduction of learned information rather than ability to solve novel problems. Self-rated memory performance was the only significant independent predictor of exam performance, indicating that other factors such as motivation, application, study skills and home background contribute substantially to exam success, once a certain degree of intelligence is reached.
Excerpts from the Article:
Abstract: Two studies explored individual differences in general memory ability and their relation to performance in a public examination taken at age 15-16 years in England and Wales. Five memory tasks were used to test both immediate memory and retention a week later. Rank orders of performance across the memory tasks, with IQ partialled out, were found to be significantly related, suggesting some general process. Separate indices of general memory ability for the immediate and delayed tests were highly correlated with each other and also with self-ratings of memory ability. Multiple regression revealed that self-rating of memory ability was the only significant independent predictor of examination performance. The results suggest that individual differences in memory ability account for between 10 and 20% of the variance in the performance in groups of above-average IQ. Possible reasons for the importance of memory in this context and implications for instructional methods are discussed.
The first study 'produced some evidence for a single underlying memory ability operating across a range of tasks. Which is independent of non-verbal IQ as measured by the Cattell Culture Fair Test...'
'Somewhat surprisingly, Cattell IQ was not significantly related to GCSE performance, but the three memory indices were. However, nearly 90% of the variance in examination performance remained unaccounted for, implying that factors such as motivation, application, study methods and home background are the main determinants of performance at this examination.'
'Both studies supported the existence of some general underlying memory factor on which individuals differ, demonstrated by agreement in the rankings over the five tasks...'
'All three memory measures (self-rating, immediate total memory and delayed total memory) were significantly related to GCSE performance. Multiple regression indicated in both studies that the only independent significant predictor was self-rating of memory ability.'
'The two studies support the existence of a general memory ability, independent of IQ, and indicate that this ability accounts for between 10 and 20% of the variance in GCSE performance. Surprisingly, Cattell IQ did not contribute to GCSE performance in either group.'
'One possible explanation is that the primary demands of this particular examination are to reproduce learned information rather that to solve novel problems. It may therefore be the case that once some threshold level of intelligence is exceeded, other factors such as motivation, study strategies and memory ability become the main determination of performance.'
23oct99b: Faxfn to Roger Greeff, Sheffield Hallam University: Selection for social work training.
Faxfn would like to ask you about the selection of students for courses on social work.
One of the criticisms made about nurse training is that initial selection process is increasingly based on "academic" qualifications and this does not necessarily measure the "competencies" required necessary for nursing.
You can see several entries in this section, which pose questions about the use of exams in selection. I understand that you also interview applicants to your courses but do not use any more formal competency assessment methods such as those mentioned in Spencer, Lyle and Spencer (see above).
These are our questions on student selection:
27oct99a: Faxfn to Elizabeth Valentine: Good memory, but no problem solver?
Thank you for your helping us with your interesting contribution.
We thought you might be able to answer a question prompted by your research: What relationship exists between "memory power" and "problem solving ability"?
A reader has suggested that, in his experience, there is at best a poor correlation between academic ability (measured by class of degree) and the ability to solve problems on the job. Has there been any research done that might throw light on this?
He also speculated that a good memory may be detrimental to problem solving because it encourages a "rote learning" mind set (i.e. Inappropriate stock solutions are remembered rather than new problems solved). Is this remotely possible?
Would this account for the claim made in the 1970s that the requirement that research students have good degrees meant that half the outstanding researchers were missed. Do you know anything about this too? We were told it concerned students that had an alternative entry to an American college (possibly Harvard).
Answers from Elizabeth Valentine:27oct99b: Faxfn to Professor Smithers: Exams and leafy suburbs
The Daily Telegraph has just reported
"UNIVERSITIES are to recruit students by postcode under a new system in which preferential treatment will be given to applicants from poorer backgrounds."and
"Alan Smithers, the professor of education at Liverpool University, condemned the move, saying: "There is an obvious danger that bright students with affluent parents will be discriminated against unfairly. Admission to higher education should be on the basis of ability, not quotas."Rising from this and other issues on faxfn we would like to ask:
Miss Sparks (see below) had some very hard words to say about Faxfn's publicity department for the advert in Private Eye. "I may have been a bit of an alky but I was no whore!", she said (rather shouted). Much grovelling on our part. We grovelled enough for her still to allow her piece to be posted. We hope the Advertising Standards Authority didn't notice.
19nov99b: Catherine Sparks: My Degree: 3 weeks work, 87 weeks drunk.
First I would like to say I am neither boasting about my performance at University or ashamed of it. It is just a fact that I achieved a degree by cramming for three weeks before my finals. Admittedly, a good short term memory plus very fast reading ability may have been an advantage, but I firmly believe a degree does not require three years protracted study. Especially in the field of Social Administration where so much of the whole course is geared around a reading list of about eight to ten "Bibles".
I did begin University with noble intentions of churning out every essay on schedule, spending every afternoon studying in the library, wearing cricket jumpers in university colours, embracing the cultural opportunities at an elite establishment, and finally graduating in the top ten of the year to loud applause. Sadly, ideals were soon compromised (on the first night at the Union bar - a binge which lasted seven days).
I revised my goals to three afternoons spent in the library and essays only matter of days late. This degenerated further and further until my whole day turned topsy turvey and I was arising at 4pm to watch Neighbours and proceed to the nearest bar and club etc. ending at around 6am. I soldiered bravely on, with no encouragement from anybody I may add, to finally graduating with a lower 2nd Class degree - such a grade well deserved you may say but I would like to add that in the entire year exactly half graduated with 2:1 and the remaining half graduated with 2:2. So I was no worse off than 50% of the serious minded studious graduates. So who was right? I had a bloody good three years and graduated with the minimum of effort and my degree had proved a vital and useful tool to further my career.
A valuable lesson I learnt from my degree was that it was possible to make the minimum of effort to achieve the maximum result.
Since graduating I have been employed by a prestigious and massive multiple retailer who held me in the highest esteem for my hard working and enthusiastic commitment. For the past five years I have run my own business which is very busy, popular and a credit to me.
I have always had a very strong work ethic, but have been disinterested in the academic and I firmly believe that educational qualifications should not be the main yardstick by which people are measured. If you can cut corners academically to reach the same goal this should always be done. I also believe that academic cheating is a good thing ( if you can get away with it) because it gets you through the educational process to areas of life and work where you can really prove your worth to an employer having overcome the "academic results" stumbling block.
Degrees are not a good measure of employability. Many graduates with "bad" degrees are excellent in the job situation and employers that stick rigidly to a "1st or 2:1 only" rule miss out on brilliant opportunities.
07dec99a: Faxfn to Professor Smithers (2): Graduates and dropouts
The Independent reports (27th November 1999):
Dropout rates at some universities are an astounding 40 per cent, according to figures to be published next week which herald the first official league tables for higher education.and
Professor Alan Smithers, of Liverpool University, was worried about taxpayers' money going down the drain. "I am startled that so many people who have gained entry are leaving in such a short period of time," he said. "It may not be good value for money for universities to be open to everyone if you then have 40 per cent of them leaving. This is worthy of systematic inquiry."Faxfn asks:
I saw your open letter to Prof Smithers. Surely you could have made your point more succinctly. Take a quote from the paper on one type of psychometric test you mention elsewhere "Competency Assessment Methods" (Lyle Spencer et al.):
... an increasing number of studies were published which showed that traditional academic and knowledge content tests, as well as school grades and credentials:
A quick internet search shows the good professor does not seem to regard psychometric tests with favour:
Professor Alan Smithers at the University of Liverpool's Centre for Education and Employment Research regards the tests like astrology.Perhaps Lyle Spencer might say degree results are like astrology too. Can Professor Smithers show us the research which shows degrees give employers the gold standard they want?
08mar03a: Higher Education Funding Council: "It is not always the better coached student that performs better."
"A-levels are an important guide to educational achievement but they are not a perfect test of everything," he said.This article also reports that Sir Howard Newby, chief executive of the Higher Education Funding Council as saying that the high A-level grades obtained by pupils at independent schools did not necessarily indicate they would do better at university than their peers who obtained lower grades at state school.
He told MPs that independent schools were good at getting high A-level grades for their pupils, but that did not necessarily mean those young people would do better at university than their state school peers: "When they come through to university, it is not always the better coached student that performs better."09mar03a: The Empire Strikes Back: Most degrees are good investments
...we cannot conclude ... that the skills that employers are actually using and looking for are indeed the ones gained late in the day. The most valuable could have been acquired much earlier - by age fourteen, sixteen or eighteen - and we have seen strong suggestions that this may indeed be the case.
In contrast to this research published by the UK National Statistics in Labour Market Trends does show graduates get, on average, good value for their investment in education but that for some types of degree, especially arts degrees, there is a negative rate of return.
On 07mar2003 The Independent had a story School leavers with two A-levels earn more than arts graduates which reported this.
Graduates of arts courses would be better off if they had quit full-time education after A-levels rather than studied for a degree, research published yesterday indicates.But the general conclusion of the research is this:
There are large average returns to education. There is a significant variance in returns across individuals. There is no evidence that the recent expansion in higher education has resulted in the financial returns falling – implying that the expansion in supply is just keeping up with growing demand. The effect of education on wages actually does work via higher productivity. However, while it has been shown that the private returns to education are large this is not enough to answer the question of the extent to which this education should, or should not, be subsidised.(Our emphasis.)
The authors, Ian Walker of the University of Warwick and Yu Zhu of the University of Kent, base their conclusion on the fact that when the school leaving age was raised in 1973, only a few extra pupils stayed on at school past the increased minimum leaving date. They consider two explanations of the relationship between earning and education:
To investigate this important issue it is necessary to look at it in a roundabout way. If people choose education in order to distinguish themselves from others then, if a low productivity group were to raise its education for some policy induced reason, the more productive would also want to invest in more education in order to continue to distinguish themselves from the less productive. On the other hand, if education simply makes people more productive then educating one group more has no effect on the decisions of others.The authors argue that because few extra pupils stayed on after the extra compulsory year, they are not choosing education in order to distinguish themselves from others (i.e. fight off the competition for jobs). This gives weight to the only alternative - that education simply makes people more productive.
What is not clear from the paper is to what extent the authors reject the hypothesis that employers pay wages according to the qualifications of their employees in the (mistaken) belief that these are a good guide to productivity.
Clearly the publication of this paper in Labour Market Trends is an important statement by one branch of Government.
09mar03b: Anton Stark: Necessary credentials at the BEEB
I left school at 18. I took three A levels but as I was moving house and starting a new job, I have never discovered if I passed or what grades I got. I am now 35 and have worked all the time since except for one month when the company I worked for went bust.
I started as a retail assistant in a photo shop. I then got a job in London working for a photographer for two years before returning North to work in a photographic studio in Leeds. After five years, the studio went bust and I started my own business, which become so successful I have saturated my particular market.
While I was working in London, I applied to the BBC for the position of trainee cameraman. Out of hundreds of applicants I survived three interviews and got down to the final dozen. I remember that there were about as many jobs as final interviewees. I was offered a place by letter shortly afterwards.
Within days I received a phone call to double check my physics O level result. I had a C grade but BBC Recruitment insisted on a B grade. They had initially misread the CV and thought I had a B. So no job. They did say I could reapply the following year with a B grade in O level physics and I would be offered a job.
Of course, I followed other avenues.
11mar03a: Found on the web: Student Admissions in California
This reports a different approach for selecting students for university courses in the US, where school grades and SAT tests are used routinely. SAT tests (Scholastic Aptitude Tests) are successors to intelligence tests, which were designed to be a measure of aptitude not of learning or cultural background. SAT tests have been recently discussed as solution to the UK's selection problem now that A-levels alone are not regarded as a gold standard.
But University of California President Richard Atkinson feels that the current SAT tests add little information to what other tests and grades show about a student's academic capabilities. "If you know the definition of those words," he said, "the reasoning is trivial."
At the same time, it discriminates against poor and minority students, and distracts attention from the core academic subjects that high-school students should focus on. "If you know the definition of those words," he said, "the reasoning is trivial."
California has made significant strides in the last few years by establishing clear guidelines for K-12 school curriculums, setting high academic standards and employing standardized tests to assess student achievements. Yet for all these reforms, the admissions process at the University of California and other universities across the nation still puts great emphasis on the SAT I, the Standardized Assessment Test, aligned neither to standards nor school curriculums.
But he approaches the postcode issue in a different way to Chris Woodhead and Alan Smithers.
But changing standardized test requirements is only a first step. We should also adopt a more comprehensive, "holistic" admissions process that takes a range of factors into consideration, from the quality of a student's high school to the opportunities available to that student. A young person who has made exceptional progress in challenging circumstances needs to be given special attention.
The American Prospect article reports a radical new idea that the top few percent of pupils from each high school should be admitted to university.
Atkinson says that UC's policy of admitting the top 4 percent of the seniors from each high school, the first of his admissions reforms to go into effect, has been a resounding success. Last year it prompted an 8 percent jump in the applicant pool, of which a significant percentage are black or Hispanic, many from high schools that had rarely sent students to UC before. In its first year, roughly 80 percent of the graduates in the top 4 percent of their high schools applied.