In need of a Base Case
Comments about coding, research and life

One of the largest criticisms of the Running on Empty report is that the states that score highly in terms of standards are some of the states that have the lowest implementation levels of those standards (as measured by CSTA surveys and number of students taking the AP computer science exam). Today, Change the Equation posted a commentary entitled that “Standards are Less than Half the Battle“.

This may be especially true in computer science, as the courses are most likely not required, and often only count as an elective credit for graduation. There are absolutely cases where exceptional standards are only partially or sparingly implemented, and with no assessment we cannot tell what is going on in classrooms.

Does this mean that standards are not important? no. They are a starting point. They express to school officials, who may not be experts in a particular domain, the important concepts in the field and the appropriate learning outcomes for students at a variety of levels. I applaud Change the Equation for pointing out that the high and low ratings from outside agencies may not reflect the ground truth. Perhaps we need to include parallel discussions about both improving standards and highlighting the implementation that would accompany classroom practices in a school where full implementation of excellent standards existed.

Subtitle: Attrition, Attribution, Self Efficacy, and Feedback

So, for those familiar with the work I am pursuing for my thesis, it should be no surprise that I believe that feedback is an important part of any educational environment. The type of feedback we choose can have profound effects on student learning, but I am also convinced that it can impact their motivation as well. This is a post inspired by a recent WSJ article that was also picked up in the NYT as well.

There are many research hypothesis that have made their way into my “folder”. I keep a folder in my desk where I jot down notes based on other works, or just ideas for research studies I’d like to conduct if I have the time. I believe that this folder will become useful if I ever want to make a change in my general research direction, or if I eventually have students who are looking for a project but not sure where to start. Each idea contains references to the presentations or works that inspired the thought and I would let my students look through for something that interested them and then have them read the work so that we could talk about their take on whatever principle applies to CS (or Math or EdTech) education. There is a very large subsection of that folder that belongs specifically to self efficacy and its impact on learning, performance, retention, engagement, and perseverance.

Computer science is an interesting discipline. Most scientific disciplines have a level that you reach where the problems you are trying to solve are not prescriptive and take a number of attempts or false starts before you are on a path to success. For complex math proofs, it is often the case that you try several different methodologies before hitting on the one that works. For natural sciences, there is a reason why its called an “experiment” - what you try does not always work the first time when you are doing lab research. CS obviously has these problems as well - but whats interesting is that at the novice level, the first course even, we are often already into that state. Students are asked to complete a programming assignment that may have an abstract connection to what was done in class, but is rarely a direct translation of the lecture. Also, the feedback is much more immediate (in the form of a compiler and test cases) than the other disciplines where you might get a problem set back the next week with a marked grade.

Therefore, in CS, a novice’s experiences are comprised primarily of something telling you that you are “wrong”. Even when you fix one problem, the tools that we use when teaching do not celebrate the fix, but present the next bug. This has to have a negative impact on student self-efficacy and in turn their belief that they are being successful in this course of study. At some point I want to study this, and what we can do to the messaging that our pedagogical environments give our students in order to help them understand that there is more work to do in order to successfully complete the program, but also to celebrate the small victories in the problem solving process early on.

Because I know a number of teachers read this blog, I’d like to include here some of the untested things I’ve used in my classroom in the past to scaffold students, and would like to evaluate rigorously in the future.

First, when introducing a new concept give small, short programs that are very similar to the lecture. Or have students code with you in class. Whether you choose to “grade” these at the same level as other assignments or not (ie. 5 point practice programs vs. 100 pt. larger projects) the quick-fix of putting together a short successful program can front load the SE of the student. Give them some early successes to build confidence before tossing the larger, more rigorous assignment at them. I often did this in a embedded way. Using Bloom’s Taxonomy I would have three graded levels for each assignment. A passing grade would involve implementing what I called a one-step. The program would be one step further than the lecture. They could type in the notes (any code I posted online would be in .gif form so they could not just copy/paste but actually had to type), and make one change. Step two would be a little harder and involve some application of other knowledge or integration of other concepts. This would earn them either a B+ or A-. In order to get an A they had to “impress me”. The last points were open ended and require that students then create some application of the program. A great example is when I taught students to use buttons and click events. The lecture would show how to create one button and have a single text box pop up when the button was pressed. The one-step was to create a program with two buttons and two messages. The next level involved maintaining a state across button presses - counting how many times each button was pressed. The last level was up to them and I got everything from touch-screen type McDonald’s ordering menus, to games, to a four function calculator.

A second way I scaffolded students within each assignment was the use of unit testing. In addition to helping me out by providing automated grading, students could see how many tests they passed and failed, and rather than just seeing the next error many interfaces will show you all tests - with passed tests in green. Watching the number of green tests increase over time is a good way to feel like you are making progress, even if there are still errors to correct.

Overall, I think SE is an important measure of just how well we are creating content and pedagogy that has the chance of engaging students and reinforcing learning.

Recently I came across the 100Kin10 website. (Yes I sent an email to Jan Cuny about it to see if she knew about it and had fit the 10K project for CS educators with it)

The underlying premise of the initiative is that in order to produce the next generation of scientists who imagine, build, design, and create the future technologies and products that will keep America competitive in the next century, we need great teachers. Students need to see STEM fields as something exciting and worth the time and effort to study. The goal is to train 100K new teachers in the next 10 years. Based on statistics from the NCES (National Center for Educational Statistics) approximately 170K new teachers start work in a school district each year(based on 2005). Most of these replace leaving teachers although a few are new hires for expansion. Even if only 1/3 of these teachers are STEM teachers, that still means that there will be 56,600 teachers each year being hired. I wondered if the 100K was a little small of a number?

The movement is just getting started, and schools of education are committing to training teachers for the program. However, as a Computer Science Education PhD who is looking for a faculty position in the fall, it is becoming very clear to me that the movement is going to be very heavily science and mathematics based. My fear is that CS departments are going to think I belong in the education school, and that there are NO CS education positions in ed schools (so I’m applying Math ed - I have a Math ed undergrad and spent 10 years teaching high school math, and CS is a Math at the K12 level in NY). If I am having trouble finding such a position, whats going to happen to our share of the 100K?

Thinking about the next round of CE21 grant applications I was thinking about how to train teachers for the CS10K project (similar idea, shorter time and only 10K teachers). It would be helpful if I could get a position at a school of education and offer courses in pedagogy and methods for either preservice teachers, or for inservice credit for current teachers. I think even either a blended or wholly online approach may be the best way to reach the teachers who would be interested in such a course. Still thinking.

Overall, this post is a little bit of a ramble - I’m hoping that some of my followers who are also bloggers pick up the movement and start making the CS Education community aware and hopefully reach out to their schools of education to find out 1) if they are a cooperating institution, and 2) how can CS get in the mix.

For a number of reasons I have been thinking about the pedagogical implications of teaching recommendation engines for the past few days. I’ve always felt that simple machine learning concepts are easy to understand and can have important motivational as well as curricular outcomes.

So, in preparation for curriculum writing I always ask myself who is my target audience and what amount of time do I have. I responded to a message today on the SIGCSE list requesting ideas for a 25 minute presentation to inner city youth about computer science, and at the same time I am thinking about the curriculum for a summer workshop for inner city girls that may happen in NYC this summer. Then I read an article about how ad companies are looking for more data people - and felt this was a nice window into the way that CS doesn’t just lead to a job a Google or Microsoft.

So, to set up the lesson I would start with a combination of what is a recommendation engine, where have you seen them (ie. amazon, shopping websites, netflix, etc.) and start to ask students how the recommendations get created. At that point we get into a little bit of what is machine learning, and how is it better than someone’s best guess, or just fixed recommendations based on product and not person.

Next, why is this such a big thing? A times article highlights that 1 in 3 people who see a recommendation end up making an additional purchase. More sales = more $$ for stores and that’s a big deal. Such a big deal that Netflix offered $1M to anyone who could improve upon their recommendation system to a certain percentage.

So, what would you like recommendations for? It doesn’t have to be a web business either, think about ways you could use your phone. (engage the students in discussion here - for the 25 minute version this would be quick - for the week workshop this would be the beginning of group project brainstorming).

So how do recommendations work? Here I would provide a basic introduction - what are features, how can we compare them? Perhaps one of the most simple things to build is a decision tree (especially if you have a carefully chosen data set). Explain how to build a decision tree and give students a data set that they can build their own trees from. For the 25 minute workshop - this is where I’d give them a slightly more convoluted data set and give them a base accuracy rate and see if they can do better. For the week workshop - move on to talking about things like prior probabilities, nearest neighbor, etc.

Now, I also feel this is a great lesson to do with your math department if you are a high school teacher. Recommendation systems are a great application of probability and statistics. Its a great setup for talking about independence of variables and different ways to look at data. (Its also a nice introduction to the CS courses and how CS is in your world)

As I get to actually writing this curriculum (ie. slides, worksheets, etc.) I’ll be sure to post it.

What are your experiences teaching this subject? Any ideas for me?

A recent post on the SIGCSE list had me thinking about the way we teach coding, but even more generally complex problem solving, in the classroom. Often the teacher will propose a problem and work through the solution with students. In some cases the solution my be complex and require not only the new concept the teacher is trying to reinforce, but also older skills that may not have been mastered yet.

The question was posed to choose between two styles: (A) work through the entire problem and talk while you work - encouraging students to work out the problem (or type in the code) themselves at the same time, or (B) share a version of the slides beforehand with students so that they do not have to try to take notes (and therefore be too busy perhaps to get some of the subtle things that you say) but it also leaves them open to being less engaged with the material.

Both options have their pros and cons, but I’d like to focus on a hybrid answer that I believe takes some of the best of both. I would often give students “scaffolded notes”. The idea came from a technique used with special education students. Students get a “mostly filled out” copy of the notes, where the blank spaces are carefully chosen to reinforce the goals of the lesson. If a new concept is being introduced - the blanks will focus ONLY on the new concept, while if the purpose of the lesson is review - the blanks will highlight the review concepts and skills that are important for the students.

The reason I titled this post “Cognitive Load” is that the scaffolding of the notes allows for the students to focus their working memory as well as the larger part of their attention onto the focus of the particular lecture. If they have not yet mastered some of the prior concepts necessary the scaffolding can help keep that particular weakness from affecting their ability to understand the new material - as much as possible. With complex problem solving situations this is very important.

In Computer Science we often hear that either students “get it” or they don’t. Some have postulated that this is actually a “beeding edge” hypothesis, and its not because some students just do not get it, but instead that they missed an important concept at the beginning and then the course just ran away from them while they struggled to catch up. Personally, I ascribe to this theory and think that by managing our student’s need to rely on mastery of prior concepts in order to even have a chance at understanding new ones (or the available working memory to process them) we need to think more about scaffolding and cognitive load in both our lectures and assessments.

I am about halfway through the new Framework for K12 Science Education that was released recently. It was not a surprise that the community reacted with messages here and here. I want to wait until I have read the entire document before responding to those comments. I am about halfway through and I have some other thoughts and comments about what I have read so far. With a document that is over 300 pages I know that there might be more than one post on this topic.

First off I have to commend the writers of the framework. In addition to nodding my head in agreement while I am reading, it is also inspiring me to think about CS curricula in new ways - and for a policy document outside my own field that is a huge compliment.

The primary take away from the first part of the framework is the idea of cross cutting core ideas. One of my favorite quotes from the framework comes on page 2-2:

Understanding the core ideas and engaging in the scientific and engineering practices helps to prepare students for broader understanding, and deeper levels of scientific and engineering investigation, later on - in high school, college, and beyond. One rationale for organizing content around core ideas comes from studies comparing experts and novices in any field. Experts understand the core principles and theoretical constructs of their field, and they use them to make sense of new information or tackle novel problems. Novices, in contrast, tend to hold disconnected and even contradictory bits of knowledge as isolated facts and struggle to find a way to organize and integrate them.

This has been a grounding of so much of the research I have wanted to pursue in my PhD studies and I hope I will continue to come back to it as I move forward to other research in my career.

So, what are the core ideas, the fundamental principles of literacy for every american to understand? The science framework has their entire set of these core principles on one page (ES-3) and looking at the simplicity of them is inspiring. For computer science - could we do the same? Would we be able within one page to describe the clear ideas in short (3-8 word headings) succinct language that any educated person could read and understand their importance?

A question that occurred to me as the framework began talking about the history of scientific development is that perhaps we are letting our own history dictate too much the curricula we put forth. For many of the faculty teaching computer science they were a part of that history and its development. You learned theory and logic and programming as core skills to be able to implement algorithms and develop solutions to problems. Eventually HCI and Robotics and a myriad of other applications grew out of the way that these solutions were constructed and launched whole new fields. We figured out how to communicate in a variety of ways, raising new issues about information, communication, networks, and security and privacy.

In many of our courses we are attempting to have students take the same path, and reading the proposed CSTA curriculum it feels very heavy in the algorithms, theory and programming side of CS. While those skills and knowledge may be important for the computer scientist - isn’t it more pertinent that the literate citizen understand not the physics of what makes the engine in the car go, but the controls on the surface, and maybe a little under the hood maintenance? Finding that balance is going to be hard.

And because its going to be so hard I’ll say it again. Finding that balance is going to be hard.

But perhaps we need to take a step back and separate out the skills we want our majors to have (not even the chemists take organic chemistry in HS) and a set of knowledge and skills (1) that our population should have and (2) that show what the discipline is about at a high level allowing our future majors to make informed decisions about entering it.

Enough ranting, musing, and thinking about this for now. I have thesis work to dive into. And more Science Framework to read.

The National Academies have released a new Book/Report entitled “Successful K-12 STEM Education: Identifying Effective Approaches in Science, Technology, Engineering, and Mathematics. Its short (about 30 pages) and a quick read with interesting recommendations.

There are two approaches I want to take to the article. First is an analysis of their recommendations and the alignment with our own goals and understanding of increasing CS education. Secondly, I would like to look at their “rating” system for effective education.

Overall the report lists 3 goals for US STEM education: (1) Expand the number of students who pursue STEM degrees and careers and broaden participation (2) Expand the STEM-capable workforce and broaden participation (3) Increase STEM literacy for all students.
These goals align very will with the CS Education goals - in fact switch CS for STEM and I dont think anyone would argue that these are not important.

I also like their definition of STEM literacy (mostly - see below). “STEM Literacy = defined as the knowledge and understanding of scientific and mathematical concepts and processes required for personal decision making, participation in civic and cultural affairs, and economic productivity for all students.” The idea of personal decision making and participation in our culture and economy are interesting tenets for literacy. What are the computer science or technical concepts which satisfy those constraints? Should the average person understand how recommendation engines work? Search? the internet? These are clearly things that they use every day but at the moment get very little formal instruction in. (especially if standards are any clue)

When describing effective STEM education the report lists key points: (1) A coherent set of standards and curriculum, (2) Teachers with high capacity to teach in their discipline, (3) A supportive system of assessment and accountability, (4) Adequate instructional time, and (5) Equal access to high-quality STEM learning opportunities. This list is a good one in my opinion. It again echos many of the things we are trying to do in CS education.

The only problem I have with the report is that it really only addresses Math and Science education. It repeatedly emphasizes hands on, engaging activities for students to get them interested early in their academic career, and yet like many STEM policy documents it ignores the two letters (TE) that offer the greatest opportunity for hands on learning and for students to see the immediate applicability of the concepts they are learning.

Take a look at the report for yourself, then come back and let me know what you think about literacy and also the exclusion of TE.

This week I spent three days at UVA attending training that is the first step towards certifying me as a reviewer for the What Works Clearinghouse for IES. I have written in the past about some of the practice guides from IES and the useful recommendations they put forth.

The training itself was great, both from the standpoint of a future reviewer, and a researcher. I now have a firm understanding of exactly what types of study design yield recognized, strong, evidence for an intervention’s success or lack of success. In particular the trainers pointed to some new references for study design, especially one on Single-Case technical documentation.

A Single-Case Design is an “adaptation of interrupted time-series designs”. What does that mean? It means that you study a single case of a classroom, a group or a student over time where you apply and remove the intervention in specific patterns with measurements at various points. This type of study could have a lot of use in the CS education community who often look to their own schools or universities and often are unable to attempt a randomized controlled trial. I highly recommend that if you are embarking on a plan of research to test an intervention in a single-case environment or without a control group that you read the documentation. My understanding is that more and more funding agencies are going to look to the WWC standards in their evaluation of proposals as well as completed research.

Now the title of this might confuse you. Certifications are not the usual thing you think of when thinking of education and high stakes testing, but a combination of twitter posts and catching up on my Google Reader (found feedly by the way and love it), got me thinking about what if high school was more like a menu of certification exams?

You could offer a minimal level for graduation, a collegiate track, a recommendation for pre-STEM college or pre-humanities. Now, don’t equate certification with “course level assessment”. One of the great things about many certifications is that they are multi-step and multi-modal and often involve so many hours of practicum work. They could require portfolio assessment and in today’s electronic age you could imagine a network where students upload projects for evaluation which get randomly assigned to teachers in other schools (if teacher A has n students, they would receive n reviews from other schools to conduct rather than having them assess their own students.) Obviously there would need to be some meta reviews and quality controls and a very good rubric for assessment, but this could help decrease the need for one-time-only performance assessments being quite so high stakes.

I can imagine more adaptive testing like the GRE where students would complete a number of problems and instead of their raw score being reported, estimated parameters of strengths and weaknesses help them target their study and redefine their learning.

Schools would need more flexibility and differentiated instruction also becomes differentiated assessment for common outcomes.

Interesting thought? I think it would be an interesting experiment to try.

So, my thesis proposal went successfully. Its what I have been working on for the past year that has caused me to mostly drop out of sight. I am exploring the role of explicit feedback regarding algorithmic abstraction during the programming process on student learning. Yes, that is a mouthful.

I’ll be talking more and more about the details as time goes on, but I really would like to get back to blogging on a regular basis about assignments, research, CS education policy and other things. Hopefully there are still a few readers out there.