--for program improvement.
As has been mentioned, I direct the National Reporting System
Support Project, which has been going
on now for almost 20 years--
working closely with state staff on data issues,
national reporting system issues, recently
with the WIOA indicators.
So I work closely with the Department of Education
on developing indicators, and putting all the requirements
in place and also most of what we do
is training on data quality and data use.
So today I'm going to run through a presentation that
gives you an overview of using data.
Why you'd want to use data?
And how we use it.
And reasons people don't use data-- so the barriers
that there are and how to overcome them.
We'll do a little discussion about data quality
because, of course, we're going to use data,
you got to have good quality data,
otherwise, it's going to be misleading.
And then I'm going to give you a little practice
to look at your own data and your state data,
specifically around the NRS table table
and show you what you can see there.
And then we'll talk about the next steps which
will be a follow-up we'll be doing
in next month in the state conference in Las Vegas.
I wonder if it's going in my other--
Pretty complex-- yeah, so it is a little complicated topic,
and so I can't really do it in depth.
I know probably like any adult ed class
there's different levels of understanding about data.
And some, maybe you don't ever use data at all
and some you're expert at.
So I'll try to cover it in an interesting way that
will be applicable to everybody, and we'll
be doing more of this, as I said,
next month in the state conference.
We're in a data driven world.
It seems like everywhere you look you have to present data.
When I started doing this work almost 20 years ago it
was an unusual thing to do data and there was a lot
of even hostility toward it.
Now, I think we're all used to it.
We all have to provide data, we know it's part of our life.
We're always taking surveys and asked to do so.
And particularly for publicly funded programs,
there's a lot of scrutiny to demonstrate your results
and to show you're doing well.
And that's pretty common now for just about every program
and particularly with education.
That was one of the first programs
to have to provide data.
You might remember things like No Child Left
Behind and other demands to show test scores
and improve performance among states.
We certainly had the NRF since-- so 2000-- so
that's been going on and now WIOA
has increased that even more.
And states also have their own requirements for data,
and we even have performance targets now.
So data is very much part of our world nowadays.
So in the-- at the federal level where I mostly work
I can assure you that data is really, really critical.
In fact, right now, in the last week, the different partners--
the WIOA partners, Department of Labor and Department
of Education--
have been looking on the latest data submission
and comparing results and seeing what's weak and strong.
So it's very important right now for federal accountability.
It supports your -- our funding.
In the past, there have been scrutiny
by Congress and other agencies, but is adult ed worth it--
money that we put into it.
And at the federal level by presenting the data
and showing our performance, they're
able to provide funding for us.
So it's been a very useful and very powerful tool for us.
May not be entirely visible when you get down
to the state and local level, but can assure you
at the national level, data are quite--
scrutinized quite closely.
Another thing about data it gives you--
a program an identity in the sense
that you can show what you do.
So people that maybe don't know adult ed, for example,
you can say, well, here's what we do in adult ed and here's--
what's unusual about us.
We promote basic skills, we help people get jobs, et cetera.
And you can show that with data.
And, of course, we're required to meet performance standards
now.
Your state, Nevada, like every other state,
has requirements to meet certain levels of performance
at every level.
And at the federal office as well the adult ed program
is required to demonstrate to OMB the watchdog
agency for all data collection and programs,
Office of Management and Budget requires every program
to present their indicators every year
to show that their programs are effective.
So-- and there's even another level.
So data is really used to identify policies and programs
that work and support claims for improvement.
So you trying to show how adult ed really is effective,
we can have that data.
And if it's successful, again, we
can show or if we need more funding
you can also use it to show the lack of coverage
or some people aren't getting what they--
performance or coverage they need.
So that's how we're use it at the federal level.
Another thing they do at the federal level
is look at national and state trends.
In fact, we just--
AIR, as the contractor to do the data analysis,
we've done some--
look at some trends for recent years, looking at enrollment
and which groups of students do better than others
and which states are performing better.
Every year the Department of Education
negotiates a performance target with every state
and uses that to evaluate performance.
And we also report to Congress every year with our data.
So the data really are that you're collecting really
are used.
Now, of course, that's not the only use of data and I hope--
one of my aims in this presentation
is to demonstrate the use--
the value of data at other levels.
So it's not just used for federal level,
but you can use it at your state.
And I'm sure your state does do that to identify areas
for improvement to measure the effectiveness
of your local programs much like we do at the federal level
and helps identify what making a difference, what
getting results.
And if there issues--
helps identify really-- with some digging
you can identify problems.
And, of course, you'll be hearing another webinar
next week by my colleague, Amanda Duffy,
who will talk about local uses of data
and how you can use them at the local level and even
at the classroom level.
You can look at enrollment trends, how your students are
progressing through test scores or measurable skill gains.
You can look even at specific classes and teachers
if your data--
able to drill down that and see what
are the effective classes, both for attendance and outcome
and also what classes are doing well and which need assistance.
So if data are so useful and wonderful to use,
why doesn't everybody use it?
And here's just a few-- some of the reasons
why it's difficult to get people to use data.
Many people are uncomfortable with it for various reasons.
And here are three that I've noticed in my work.
There are justifiable fears in using data I think.
Often data are used--
people perceive it used against you, particularly in education.
It's-- oh, this school is not doing well,
these students aren't doing the way they should be.
And it's often used to--
by those who have an agenda or vendetta
perhaps to look in a negative way, to make you look bad.
It could be used out of context.
There's the old adage that there's lies, lies, and damn
lies from statistics.
And so-- context it can be misleading
and sometimes people know their data are not very good
and really shouldn't be-- being used.
Another set of reasons have to do with your use--
your norm freezing using data.
I mean, if you're not used to using that,
if your program doesn't really pay attention to data,
you're not going to be able to use it.
If you think it's a waste of time.
If you're only using it for reporting purposes just
to fill out forms.
Or if you don't really know how to use data,
you're not going to want to use it-- not
be motivated to use it.
And-- same way--
if you can't get your data, this has
to do with structural barriers, you can't use it.
So if you were able to put data in your system, for example,
but not get reports out, you can't use the data.
So you're not going to use it or you can't get it
in the form you need it.
Another-- other things that are structural barriers
against using data are the data aren't timely.
So if you're not entering it in a timely way,
if it's last year's data, last month data,
it might not be useful to you and often that takes time.
And in adult ed programs are very stretched
with resources and staff.
So these are all barriers that can be overcome,
but they are some of the reasons why we don't use data.
So let me just look a little bit more closely at that
and why-- how we can overcome some of these barriers.
The first thing we call technical barriers, which
have to do with your database.
So-- and there's different levels
of access you can have to your data.
If a system that really has no access
or very few where you can get your data out,
that's the worst possible world.
Another way is that you have to go
through your state or another person to get the data.
That makes it more difficult because it's not timely.
The best way to do it is if you just can run a report
and go right on and get your report.
And I believe you can do that for some of your reports
with your system in Nevada.
Another thing you need to have is timely data
which speaks to entering your data in a regular basis
and not letting it get behind.
And then also having the reports you
need, if you're able to run-- go and pull down from a menu
or to find a report you need.
So all of these things--
if your database isn't able to do that
is what you need to get them to do it.
The other thing is the data collection is not an easy thing
to do and it's not something that's
a priority a lot of times.
So you want to have staff that's really--
that's their job or at least one of their jobs
to collect the data and perhaps analyze it.
You've got to all be collecting it
in the same way with common measures and definitions.
I think that's easier now under WIOA,
since we have common definition, which we haven't in the past--
and common forms and common ways you're entering the data
and collecting the data.
It's very easy to have errors with data checking-- with data
entry, so you need to have systematic error checking, run
reports, or people looking at the data,
and really staff have to be trained.
So it's not-- you can't just set up a system
and say go ahead, let's enter the data and go ahead
and do it.
You've got to have these five things really to get good data.
The other thing the promotes data use
is having leadership at your state and your program.
And I think this webinar, for example--
of how your state really does value data and the use of data.
And when that tone is set it tends
to promote data use and particularly
at your local level to-- if you're local staff
is doing that, looking at data regularly, getting meetings
around data, getting ideas about what's working and what not,
and discussing your results.
I know some programs that do that on a regular basis.
And that helps people get more get more invested in data,
see the power of data and pay more attention to it, which
results in better quality.
We even have-- we've promoted developing
communities of practice around data and some states do that.
We did a training on that a few years ago,
if you're interested in looking at that.
And, of course, attitudes have a great deal of--
to do with collecting data, particularly
at the programmatic level.
Here's just some common attitudes you can have--
that people have that are negative.
I don't have time.
We know our data is good, and we have a good system,
we don't need to check it.
I'll let someone else I'll do it.
I don't have time.
I don't know what I'm going to--
I don't have the data, I'll just make it up, who even cares.
No one pays attention anyway.
So these are all negative attitudes
that sometimes come around data that we want to dispel.
So-- and we do that by using data and finding it useful.
So I have a little poll I'd like you
to do and think about those three areas-- technical issues,
that is how does your data system work?
Does it allow you access to data?
Do you have enough support at your local and state level?
And do you have any attitude issues in your programs?
So just an anonymous poll here.
If you want to rate on one to-- on a scale of one to three--
it's not an issue, you need to improve,
or it's a serious problem.
Just out of curiosity to see and to-- before
you move ahead why don't you answer this poll.
I'll you give me a few more minutes.
So it's looking like you don't have
too much of an issue on leadership
or not so much on your data system either
or doing pretty well then.
Need to improve about 40% of you.
A little bit less saying you'd like to have a better
technical-- maybe technical issues, like get more access--
easier access.
Maybe a little more better leadership
or more state or local activities
around using data and about the-- little bit more
about attitude.
And that tends to be--
I see the highest one with--
rated two, which is normal.
I think there are a lot of negative feelings about data
collection.
So we hope to help try to dispel those.
All right, so let's move on about data quality.
Now, data quality, of course, is really important.
If you don't have good data you're
not going to be able to use your data.
And it's going to give you--
worse than that it's going to give you misleading results.
So I just would like you to look at the full list of definitions
of quality and what you think is good da--
what is good quality?
So if you can just type in the chat box, pick one of the--
one or more of those and write down what you think
is quality data?
Just give you a few minutes to do that--
with quality.
How do you know when something is high quality?
So think about, for example, websites.
What are high quality websites?
Let's say that might be usable.
What about your home?
What's a high quality home?
It might be safe, secure, comfortable.
Your car?
Reliable car, but easy to maintain.
So things like that are what we think about here.
So we have a couple?
There are usable, maintainable, reliable, accurate, durable,
have a lot of reliability things there, yes,
and that's definitely important.
Accurate, reliable, usable, OK, great.
Quality data, accurate, relevant, stable, functional,
usable.
Right.
It can-- that's true.
If you're not using the data, then why do you have it?
So we want you to start using it.
Usable, yes, OK.
So those are good.
Thank you very much for that.
Let me look at what we talk about in the--
specifically about data that we think about quality.
It must be objective to use data well,
so in other words unbiased, complete,
not tainted in any way.
And then integrity, in other words,
it's not falsified, doesn't have any made up date numbers,
not corrupted in any way, transparent--
basically we know how it's collected
and who's collecting it?
It's easy to understand and interpret.
Reproducible, meaning that other people are doing the same thing
will get the same result. So poor data has--
doesn't have that quality.
And utility, a lot of you mentioned that as well, it's
got to be useful.
Supposed use it, we want it to be--
and it's good quality--
part of that makes it high quality.
Here's a video, now, I'd like you
to look at for the impact of poor quality.
Hi, this is just a quick real life example
of some of the problems that can result from data quality
issues.
It's a true story.
This is Valparaiso, Indiana and the red cross--
the red square is 1108 Chicago Street, the home
of a Mr. Dennis [INAUDIBLE].
Now, he says his home is badly made
and drafty, it's round-- it's worth around $121,000.
Early in 2005, however, somebody, they don't know who,
went into the county's systems to accidentally changed
the assessment value of his home to $400 million.
Later in the year, a somewhat surprised Mr. Chanetsky
received a property tax bill of a little over $8 million.
He called.
He asked if he could pay in installments.
They laughed, corrected the figure
and sent him a new bill for around $1,500.
So far so good.
Unfortunately, in the meantime, that $400 million
figure had been used by all of the different cities
in the county, in particular it was
used to determine the budget.
The result was that, at end of the year,
because Mr. Chanetsky he hadn't paid his $8 million,
they were left with a huge hole in the budget
and had to implement emergency cost cuts.
In particular, the school district
had to return $2.7 million, which
resulted in the elimination of all optional activities,
including school trips, sports, new books for libraries,
everything.
This resulted in some very grumpy children,
some very grumpy parents, a general scandal and resignation
of the people responsible.
There are two morals to this story.
The first is that a single bad value
can very easily get into different information systems.
The second is that data quality can
have real human consequences.
So please, for the sake of the children,
think about data quality in your next project.
So there's an example of what happens
when you have bad data quality.
I have another little story to tell you about that is--
I was in a meeting last week, I think,
with the Department of Education and there was
an error on the data we were looking,
but we didn't know and the director was
very upset as to why the data was so bad, and what it meant,
and we had planned a big analysis,
and actually to make a new training on people
to show them how to correctly enter
the data for this particular element.
And a few hours later, the person, very embarrassed,
who did the analysis, said, you know what, I made a mistake
and I put the wrong data in the wrong column.
And when he corrected it there was no problem at all.
So we almost went down a similar thing, a very costly path
of doing a bunch of--
a lot of data analysis that would have been expensive
and planning and training around something we didn't need to.
So that's an example we just had last week
here at the federal level.
That reminds me of this.
So let me ask you, how would a mistake
like this look in your program?
What problems can arise?
And maybe has this happened to you?
So if you want to contribute, write it in the chat box.
What a mistake like this might be or if it happened to you,
what happened?
So a story about that.
No one?
Anyone want to comment on that?
OK.
So Ken put a promotional story in the newspaper
as well and this information.
Cynthia will be on it ACAP since the data is very data driven.
OK.
Anything else on there?
Angela.
[INAUDIBLE] students list today's
date instead of birth date.
Yeah, that's a common one is birthdays.
I've seen birth dates with participants are 110 years old
and things like that.
I can think of mistakes like enrollment,
or performance measures.
Inviting the wrong people to HSC grand ceremony.
Yeah, that would be bad.
OK.
Let me move on.
So anyway, data quality, that's the idea.
I'm wondering what you all think of your data quality.
It sounds like you're pretty confident in it.
Let me bring in a pole up, I believe.
So I'd like to know-- you know, its sounds like you
think you're pretty good data.
You can just make a guess or give your opinion in due data
high quality app.
Some of it is, not sure, or no.
Go ahead and click what you think.
OK.
So a few more responses coming in.
I'll wait a second more.
All right.
So I think you can see that.
Most people think-- have 50%, eight of you out of the 14
that voted think you have pretty good quality data.
Yes you do.
And some, only one person not sure.
So that's very good news for you and I
know your state is working very hard.
Ken Zutter, Nancy Olson and the others
are working very hard to have good quality data, Ken.
Ken's replacement, I'm sure, will be doing that as well.
So that's really good.
As we move on to our next section because I'd
like to talk now about using your own data for program
improvement.
OK.
So how do you do this?
It's not an easy thing to do.
It does require a little bit of training.
I'm just going to give you a little bit
of training right here.
And it's easier than most people think.
Although, it does take a little bit of effort
and a little bit of thinking.
Basically, I tell people when they
ask me, well, how do we look at data, I sit down with them
and say, what do you want to know?
What do you want to know about your program?
And then put it in the form of a question.
And that takes a little bit of work
because it has to be a question you can answer.
So think about what it is that you want to know and develop
a answerable question around it that's specific.
Then you can look at your data to answer that question.
So it might be all of the different levels of EFL levels
seen at different levels.
Are they performing and getting MSGs at the same rate?
And so that you can look at your EFL breakdown by level.
And you can go answer that question.
Then once you have that answer, you
can discuss your findings with your staff.
And then you usually read out the program improvement.
And those are the four simple steps.
Let me go back.
OK.
What areas might you want to look at data?
I generally find two areas that I
like to focus on because they're the most successful
and the ones you can get data most easily.
And we collect it in a pretty well, reliable, and valid way.
And those are, attendance measures and MSG,
you know, mostly test score-- pre- and post-test scores,
although under WIOA expanded that a little bit.
So let me ask you, what questions
can you answer about--
would you like to have answered about attendance and MSG?
What do you think your data can provide to you?
The data that you have that you collect.
Anyone.
Just type it in the chat box.
OK.
Got some good questions there.
Correlations between attendance, times of day,
optimal numbers of weeks, pre and post percentages,
class size, CASAS scores aligning with MSG,
with TABE and the NRS level.
OK.
So great questions.
Great things for researching and great things
that you might want to use your data to ask.
Certainly things you can use for program improvement.
For example, if you're finding relations between attendance
hours and MSGs, you can adjust your services perhaps,
the number of hours you provide.
You can find that optimal number for particular types
of students and other things that you've mentioned.
Typically, some attendance questions,
are students staying?
And which type of students are staying?
So retention is another issue you can study,
looking at attendance hours.
And here's some of the data.
Somebody asked, you know, what's the best
way to look and collect this.
You can look at your contact hours student's getting,
intensity and duration.
So we could make a measure, like,
how many hours per week and what type of instructions
being provided.
And that way you can look at which materials are relevant,
are working, motivation of students,
assuming that higher attendance means more motivation and more
retention means more motivation, and maybe correlate with goals
if you have data on student goals,
whether they're being achieved and how many
hours are needed for that.
So attendance really can be a very powerful thing
to look at and help you with program improvement.
Let's look at MSG now.
Basically, pre and post testing scores, as well as
what the people are getting at high school equivalencies.
And basically the questions you can ask there are,
are your students--
Look at the table one, which has just
been race and gender by EFL.
So what you're able to look at there is those demographic.
Table two is age assisting and gender.
And three has program type and age.
I have an example of another state.
This is not Nevada.
I don't know how well you can see that on your screen.
But what this shows is educational punching
level and the ethnicity groups, and whether they're
male or female.
So that's a very simple table.
And what you can look at there is differences
by gender and differences by race and ethnicity.
So in looking at the table, I hope you can see it.
I'd like to show you there for one thing you
can see there is that in AB and ASE,
there are more male students, except for Asian.
And then EFL there are more female students.
And that's a pretty common finding,
that women tend to take EFL classes more than men do.
And men tend to be more in the AB ASE side.
And so you might wonder, why is that?
And that's a finding you can pretty consistently see.
And what does that mean?
I don't know if anyone would like to comment on that.
Certainly that's a common finding.
Another thing that you can look at table one
are differences by ethnicity.
And not surprisingly, in this particular state--
knowing what state it is.
It's actually for Florida.
There's more Latino EFL students.
We also see in Florida that most students are low level.
So for ABE, most are on ABE levels two and three,
but yet for EFL they're higher level--
three, four, and five.
That would be an interesting finding,
if you were the state and wanting to know why that was
and how you wanted to reprogram your--
if you wanted to maybe increase the levels of maybe
lower level EFL your students, which tend to be more in need.
Why aren't they attending at the level you might expect them to?
Want to point out though, I don't know if you can see it,
but this is typically the case with smaller
states in particular, is sometimes
you have very small numbers.
And when you have very small numbers,
you can draw a very close conclusion.
OK.
So now what I'd like you to do is take a look
at your own table from Nevada one, two, and three.
Nevada table.
And I believe you can get them in the handout box.
Winnie, if you could help with that.
When you look at each table I want you to consider what
are the data saying for you?
What are they telling you and not telling you?
What do you think it means?
And we know that particularly when you do data analysis,
that's not the end.
You usually have more questions than you started with.
So what other information would you
like to know to understand that?
And did they suggest any changes?
So I'm going to give me a few minutes to write up
your observations and thoughts.
And take a look at tables one, two, and three, which
you can access from the little box below
or you can look at them on the screen as well.
I think we'll show them and put them up.
And there's table one and two.
And if you scroll down, you can see three as well.
So take a look and write a few suggestions, maybe one or two
for each table.
And write them down.
And one thing you should--
yeah.
And then we'll talk about it.
So we can open up a chat box there and write them down.
Or maybe we'll just have a verbal discussion about it.
But let me just give you a few minutes
to look at those tables.
If you can't see them on the screen, open them up.
Upload them and read them.
It might be easier to look at if you can look at them
by yourself on your computer.
And thank you for opening the chat box.
So write down an observation or two about each table.
OK.
I see several of you have noticed
one of the main things we see in Nevada is, we have 8,365.
If you look at table one on the bottom right,
you'll see the total.
And quite a large number of them are Latino.
And if you look at table two.
And Hispanic is our--
Hispanic is our primary group.
There's a lot of females.
As I mentioned, there's a national trend
that you see everywhere, that most EFL students are female.
And you see that here in Nevada.
There is, yes.
Ken noticed that it's a large AB level three.
Which is interesting, I guess.
That's your biggest AB population.
Anything else?
Yes.
No one's enrolled in--
we have, you don't have very much--
if you look at table 3--
IET.
Or section 243 programs, you don't have any at all.
I guess-- imagine that's a new program
and hadn't gotten started yet with it.
IET is relatively new to many states.
But you do have 182, which is probably
a couple programs doing that.
And Ken says it's the first year of IET.
And that's pretty typical in many states
because it's a new thing under WIOA
And one thing that department's pushing
is to bring those numbers up.
So you'll be seeing more about that.
From the federal level, sort of a policy initiative.
Anything else?
How would we characterize Nevada?
It's mostly Hispanic female, 25 to 44.
And among the AB population is most people
at the intermediate level.
All right.
So any reflections on that?
Anyone want to write a comment on that?
Yes.
This is you most recent data that was just submitted.
I think Ken was saying it's 16, 17 data.
It just turned this data in a couple of months ago,
a few months ago.
And it's what's being looked at right now at the federal level
for each date.
OK.
Any surprises?
Any what you expected?
Any questions that maybe--
one thing that if you really get into data,
which is fun about it and gets exciting about it
is, it raise a question like why this happening?
Perhaps what are the outcomes of EFL
compared to ABs, since that's such
a big part of your program.
Which particular levels are performing better?
So those are some questions that can come to mind.
I don't know if any of anything strikes you,
anyone looking at the data.
But was this what you expected.
OK, Ken Zutter says, I'd like to compare male female to previous
here.
Are males working now?
One thing that struck me too is not too many people
were in higher level, ASE levels, 5 and six.
And I'd be curious to see are we looking at table four soon.
How many high school equivalencies
you're getting out of there.
And that's on table four, which we'll be talking about next.
OK.
Just waiting for Ken and and Nancy, who
are busily typing right now.
Please feel free to type or chime
in if you have any other comments.
Ken's saying we're seeing MSG gains from HSE and mid AB
levels.
That's a trend we see in other states
too, that people come from all levels
when they get their high school equivalency.
It doesn't have necessarily-- or isn't
even very often in AB and 5 and 6, which
is where you'd expect it.
And did analysis once, years ago,
where we found that students get a high school
diploma and high school equivalencies
from just about every NRS level.
Of course, mostly at the high levels,
but even the people at level one will
get high school equivalencies.
And Ken's saying, that it's nice to be able to get MSG from HSE
now.
Yeah, that's a new thing we added at the federal level.
New requirement to allow high school equivalencies
to be counted as measurable scale again,
whereas before they were not.
So it helped a lot with the performance.
Incidentally, I'll give you a little preview.
The adult education looked very, very good this year
on our performance on MST compared to our partners.
And I'm sure there's a lot of gloating
going on at the Department of Ed right now about that.
So good job to us and specifically
to you at the local level, helping our MSG look good
at the federal level, which as I started off the presentation
today, it's very important in these times of budget cutting
and reduced resources available to us,
it's really good to show that we are performing.
And even in many cases, better than our federal partners
under WIOA.
Not that we're in competition or anything, but the way it goes.
OK.
Let's go on.
When you do look at data, remember
to think about quality again.
And if something doesn't seem right in data, a lot of times
it's not.
And you should definitely question that.
One thing it's very good to be, if you're
a data analyst, when you get into data is to be skeptical.
To always question what you're looking at.
It's very easy to misinterpret data
or to see what you want to see in data.
And it takes some, you know, a little bit
of objectivity and thinking to really look
at data in a fair way and a way that's really helpful.
And certainly quality is something
you should always think about.
And particularly if you're at the local level, when
you know of how your program is collecting data.
If you know of some anomalies and things you maybe
aren't doing right, or things you could do better,
you might be a little skeptical about it.
And even all data.
How is it being collected?
You see polls a lot of times, for example, in the news.
And, you know, that 80% of people
think this and 6% of people think that.
The first question you should probably ask is, the quality.
Where is that number coming from and is it really trustworthy?
And it's always good to be a skeptic when it comes to data.
We want to look at the averages and the ranges.
A lot of people ignore variation, ignore ranges.
In other words, what the high and the low number
is and they look at the average, but sometimes
that can be misleading.
So while it's important to look at the average,
you want to look at the high and the low.
And sometimes you can get some insight into that.
And certainly, sometimes just looking
at a number in isolation doesn't tell you very much,
but when you look at it in comparison to a standard
or to a different group, it will have more meaning.
So if I just tell you--
for example, I just said, you know, we look good compared
to our federal partners.
I think our national performances on MSG
is around 44%.
And if I just told you that, you wouldn't really know
or you might have an opinion on whether that's good or bad.
But then when you look at another provider
or if you look at labor, which is considerably lower,
you'll say, oh, well we're doing good compared to that.
Or you might want to look with these with your local data--
you might look at different programs or different EFL
levels.
And that's when we look at table four,
you'll be able to see that.
OK.
The next step is, I'm going to conclude very quickly here,
but I'd like you now to work on your own
and do that in the next month or so.
So I'm going to give you a little assignment.
If you are going to the conference
next month in Las Vegas, I'd like
you to look at the table four for your program.
And I have here in the handout, the statewide table
four as an example.
But what I'd like you to do if you are coming
to the conference, and even if you're not,
you might want to do this just to get
into the habit of looking at data-- and as I said,
that's really one of the keys to using data,
is to do it with your colleagues and do it collaboratively.
And just take a look at your data.
Don't just enter it in the computer
or collect it and forget about it.
And kind of the key to using it and understanding it because it
becomes like a negative cycle if you don't do it.
If you don't look at your data, you don't use it, you value it.
If you don't value it, you don't collect it well
and it becomes poor quality.
And then you can't use it.
So it's like this vicious cycle.
So by looking at your data, which
is what we're hoping you'll do here,
you might get more vested into it and look at it.
And table four is our big table among our NRS tables,
because it has both attendance and MSG
and it's broken down by level.
So you can really look at what's going on in your--
particularly if you look at the local program level.
So what I'd like you to do that, is get your table four
for your program.
Collaborate with your teachers.
And next week when we do the teacher webinar,
we'll be giving them the same assignment.
So you might want to do that to take a look at your program
overall.
So pick a topic.
Either look at your attendance or look at your MSG
or do both if you want.
And come up with a question.
As I said, I think good data analysis
begins with a question.
With what do you want to know?
And putting it in the form of a question.
For example, what gains are you making?
Are your students making?
Does it differ by student type?
Does it differ by educational functioning level?
And what are you seeing there?
So look at the data and then try to make an interpretation.
What is the data telling you and what do you need to know--
what would you like to know further?
And the same with attendance.
Like, you came up with a lot of good suggestions earlier about
looking at attendance and different ways
to look at attendance.
So take a look at table four.
You can also look at attendance patterns
and how they vary again by to student type and student
groups.
So here's table four.
This is just an excerpt of it.
I don't know I can show the whole table on there.
We pretty much can't.
So just to get you familiar with table
four, these different columns if you
haven't looked at it before.
It's broken down by the six ABE levels and the six EFL levels.
And you have the number enrolled at each of the levels.
And you'll see the total adds up to what
we looked at the other table as we were supposed to.
You see you have 8365 students last year
in your state in Nevada.
And how many attendees are--
rather, how many hours of attendance
do you have by level?
And this is the total number of attendance.
So if you divide that to column C and column B,
it'll give you an idea of what the average attendance is.
And then the other columns show the achievements
by the different levels.
So you have the first one being the number
who attained an EFL, at least one
educational function level gain.
Column E showed the number who got a high school equivalency.
And then F is the number who separated
before receiving a gain.
Those are basically your student who
left before they were post tested,
or they were post tested and then they didn't show a gain,
or they didn't get an HSE.
Column D is the number who was retained in your program
at the end of the year.
Those are people still attending,
but they haven't yet got a gain.
And then in I, you have the percentage of MSG.
So for example, let's look at level 3 AB, which
is your biggest ABE level.
You saw about a 40% improve MSG.
And your biggest one--
I don't know if you can see us down there.
The group of students that's showing the biggest MSGs
are EFL level two, which is about a 58% gain.
And the remaining columns are periods of participation.
And I'm sure you're all familiar with that now.
You might look at that too because what's interesting
about periods of participation is
those are students who left and came back
90 days or more later.
And that was a new thing, as you know, that we do for WIOA now.
That was a new requirement.
And by comparing that number and column I with column B, which
is number enrolled, you can see how many students came back,
of the same students.
How many did return?
And so for example, if I pull again
at your level three, which is your biggest level in ABE,
there were 852 participants and 864 periods
of participation, which means you had only 12
didn't come back after more than 90 days.
And that's kind of a pattern we're
seeing all over the country, is not
that many students come back once they
leave in a single year.
And it's only averaging about 3% to 8% nationally.
And I think it's pretty low here.
I haven't calculated the overall percentage for you.
But if I'm imagining it's only in
the single digit percentages.
1% or 2% 3%.
So that's something you look at as well.
And then J and K are the same--
the additional MSGs that are achieved.
So if you look at columns D and E--
so let's say for ABE level one, you'll
see there are 30 gains and four high school diplomas.
34 EFL games and 4 high school diplomas achieved.
And that's 34 out of the 78, which
translates 43.59% as you see in column H. But
if we look at MSGs for those repeat students, which
they were only three.
81 in B and column I, and 78 in column B. That went to 35.
You can see the performance didn't really change any.
So and if we look at level K compared
to level H, the percentages in those two column,
which is the left column K and H is two columns over,
that compares participants with periods of participation.
You can see there wasn't really very much
different in any of the levels.
And if you look at the very bottom, the grand total.
You'll see it was only in their state,
42.1% MSG for your whole state for everybody.
And doing it by individual participants
and doing it by POPs, it's 41.9.
So it's pretty much the same.
Very interesting for us at the federal levels since POPs,
periods of participation are a new thing.
So as you can see, there's a lot of--
I'm just trying to illustrate some things
you can look at here.
If you look at attendant hours, average attendance hours,
and performance, a lot of the things in table
four to look at.
So please do pick a--
work with your colleague and write a question or two down.
And just get some practice.
And what we like to do when we do our face to face training
in Las Vegas is ask you to report on that
and see what you're finding.
And then not only just report on the findings,
but try to go a step further and say, you know,
why do you think that is and what other information you
would like to get.
Because as I say, usually when you
start doing data analysis you end up with more questions
than you started, or at least if you have
a curiosity about data, you might want
to then go to the next step.
And it's like, well, why are students at that level,
you know, three doing so much better than at level six,
for example.
And what are the areas?
And also, remember what I said about being about quality
and do you trust these data?
That's always a question you should always
ask when you look at data.
So I think we're just about at our end here a little
bit early.
So we have in your handout, you'll
see the handout assignment bottom at the box there.
So I've written down the assignment.
So please look at that and I hope you will do it.
And I hope you will come to our Face to Face training
next month.
Remember, pick a topic.
Get a question.
Look at your data.
Be data analysts.
Tell me if you trust the data.
Tell me what else you'd like to know
and other data source you'd like to do.
And what would it mean for improving your program?
So for example, if you find differences
in one group not attending as well,
if you find performance differences,
what can you do about it?
And we looked just now at the state level, state wide totals.
So in your program you might find a different pattern.
And that might be another thing you might want too,
is look at it compared to the state if you really
want to move ahead.
But right now I'm just asking you to look at your own data.
So it may be different.
It may be the same as what you just saw at our state level.
Larry, there was a question that prompted some discussion
on the right hand side chat pod.
Susan had asked, what's the difference
between table four and 4B?
And then you can see there were a couple of responses.
But I'm wondering if you want to describe
that difference for Susan.
And if you scroll the chat pod you'll
see some of the comments in relation to that.
But how would you respond to Susan's question?
What's the difference between four and 4B.
I'm not seeing anything in the chat pod.
But table 4B is only students that are post-tested.
So if you were--
pre and post-tested participants,
they go on 4b, whereas four is everybody.
Whether or not they were post-tested.
So that's the difference between the two.
And what you would probably see--
you can certainly look at other tables.
I just gave you a simple assignment.
But if you'd like to do that and look at some other tables
when you come next month, please do that.
But you'll probably see in 4B--
I don't have it in front of me for your state.
That usually the performance is higher
because you're only looking at people that are post-tested,
participants that were post-tested.
So if somebody is not post-tested, of course
they can't show a gain.
So you get water it down.
So 4B is just to look at post-test only.
And Maryanne, I don't see anything in my chat pod.
So if there's another question--
There's a few items.
I'm scrolling for questions as well.
And Larry, because you're a host you'll see two tabs.
One says, everyone and one says, host.
So that might help you see that.
Ken had responded, a table four performance
includes high school equivalency MSG, which is not shown on 4B,
as well as a few other comments.
Right.
Four has more data, of course.
4B is only for post-tested.
It's only pre and post-testing.
That's right.
But really I don't see anything in my chat pod.
I don't know why.
OK.
That's fine.
I can scan for questions.
Are there any other questions from the group?
I see TMCC team is typing.
As well as Ken.
Words of thanks, Larry, from a couple of people.
[INTERPOSING VOICES]
TMCC says, I was unable to download the handout.
Can that be sent?
Yes, we can send whatever.
We'll send to the group.
Angela Holt is typing.
As well as Winnie Barber.
Thank you, Winnie.
You'll send out the handout.
That's great.
And Angela is our last person typing.
You're welcome, Angela.
And, Larry, are there any concluding comments
I'm not seeing other questions.
I don't know if there's other--
there's still an opportunity.
Any questions you have.
Larry, any final comments?
And I will point out, as I wrote in the chat pod and hope
everybody, that you can see it-- you're typing in the chat pods
so I assume that you are our guest can see it.
The assignment is for the March 7th and 8th Directors meeting.
So we aren't having the conference,
but we are still having the directors meeting.
And I'm going to be sending out more information subsequent
to this webinar.
Larry, any concluding remarks?
We are very grateful for your presentation today.
It was excellent substance and content.
And we're looking forward to seeing you
at the directors meeting for the follow up to this.
After today's session, I'm also going
to send out a reminder that we do have the February
8th webinar that's going to be geared towards instructors
and instructional leaders and using
data for classroom practice.
And Ken Zutter writes, "Agency data manager
should attend teacher data webinar too."
And Ariana agrees.
So by all means everyone, I think
there's plenty of room in the February 8th webinar.
Whoever you send, we are happy to include.
So please don't hedge.
You can send whoever you want to the February 8th webinar
if you think that will be of benefit.
And Larry, thank you.
Do you have any concluding remarks for the group?
Thank you, Maryanne.
No.
I just want to say, I'm looking forward to seeing all of you
or those of you that can make it next month in Las Vegas.
And we're going to start our session
by having you report on your table four and any other data
you want to bring in.
And we'll have a discussion about what
you're finding in your program.
And I'm available to help if you would
like to talk further about this, if you want further resources.
As I said, we've been doing this for many years,
working with state programs about using their data.
And so if you are interested in more in-depth--
I know there was a lot of content here.
If you want to talk about data or just
help with your assignment, feel free to email me.
I've also included Amanda's email,
who's going to be doing the webinar next week.
So it was really my pleasure to do this.
And I really like doing this kind of work.
So looking forward to seeing you all,
whoever can make it next month.
And thank you very much.
Thank you, Larry.
And thanks to everyone.
Look forward to chatting with you soon.
All right.
Well, bye now.
For more infomation >> WATCH LIVE: CBC Vancouver News at 6 for Tuesday, November 27 - Duration: 1:04:02. 
For more infomation >> WATCH LIVE: The National for Tuesday, November 27, 2018 — GM Response, Insulin Pumps, Northern Power - Duration: 1:04:40. 
For more infomation >> Tucson man continues to battle for late husband's benefits - Duration: 1:52. 
For more infomation >> Four teens arrested for breaking into home - Duration: 0:27. 
Không có nhận xét nào:
Đăng nhận xét