JOURNAL ISSUE 11

2004/2005

 

 

FORMATION OF THE SYSTEM OF EVALUATION OF

SOCIAL WELFARE PROGRAMS IN SLOVENIA

Dr. Nino Rode, FSD
Mag. Barbara Kobal, IRSSV
Dr. Liljana Rihter, FSD

 

Introduction

 

Evaluation of social welfare programs is increasingly becoming a pressing issue in Slovenia also. In 2000, the National Program of Social Security in Slovenia until 2005 was issued in which it is proclaimed that the state should assure the operation of the integral system of social security. A part of this task is also quality control of the services and support of development of the organizations which will supply the services needed to satisfy the diverse needs in the area of social welfare. To perform this task well, reliable and accurate information is needed about operation of the services and their efficiency and effectiveness. This sort of information can be produced by evaluations of the programs. This is why in 2003 the Social Protection Institute of the republic of Slovenia was authorized together with Faculty of Social work at University of Ljubljana to develop a system of evaluation of programs of social security.

 

In this paper we will present a project designed to set up a system of evaluating the performances and outcomes of programs of social welfare. The purpose of the project is to develop a system of evaluation of the programs related to social welfare which will make it possible to compare the effects of (similar) programs and give to the organizers a feedback upon which they can steer their work. Such a system of evaluation should make it possible to assess the level to which the goals of a program have been achieved, as well as evaluate of the goals through the eyes of all the stakeholders.

 

In the first part we discuss what place an evaluation should have in a social welfare system. In second part we present a typology of social welfare programs which should be sufficient to describe all the programs in this field for the purposes of comparison within evaluations. The typology is based on aim dimension from prevention to correction and duration dimension from momentary to permanent. Next the criteria for program evaluation are discussed: the quality of life, the strength perspective, normalization, functional capacities, and therapeutic effects. In the end the cost-procedures-processes-outcomes model is presented as a chosen basis for the evaluations within the system, and proposed evaluation scenario is discussed.

 

As mentioned before, in the National Program of Social Security in Slovenia until 2005 evaluations are explicitly determined only as a way of monitoring the effects of the nongovernmental programs, which are co-financed by the state. Another goal of the National Program related to the evaluation is to promote legislation, and to establish social inspectorate and program evaluators. Nevertheless, in the Plan for Implementation of Guidelines and Tasks in the National Program of Social Security in Slovenia until 2005, one of the key tasks of the service providers of public services is stated to be assuring the mechanisms for continuous assessment of the quality of the services and for evaluation of the effects. The Institute of Republic of Slovenia for Social Security should occasionally perform cost-benefit analyses of individual programs. The providers of services supplementary to the public sector are obliged to establish a system of regular monitoring and evaluation of the service quality.

 

The system of evaluation should make it possible to use a unified model of evaluation in all social security and welfare programs in Slovenia. Such a system of evaluation of programs of social security will only make sense if the results of evaluations are useful and really used in decision-making about the evaluated programs. Evaluation should show to what level the goals of the programs were reached, and at the same time offer an evaluation of the goals themselves from the points of view of all stakeholders in the project. In short, the system will make sense if its results are useful. As Patton (1997, 20) points out, in every evaluation there are many potential stakeholders and a vast area of potential uses. He advocates utilization-focused evaluation, where a shift should be made from general and abstract (potential public and potential evaluation users) to real and specific (real intended users of evaluation). Instead of playing the role of impartial judge, an evaluator has to induce and support deliberation and decision-making of intended users of evaluation.

 

The system of evaluation anticipated by the National Program of Social Security until 2005 and Implementation Plan of Guidelines and Tasks of the National Program of Social Security until 2005 is based mainly on self-evaluation. This is in line with the orientation of modern evaluation research, which tries to capture the evaluated object as a whole within its context, and thus makes possible informed evaluation and decision-making.

 

In this context Patton suggests using a criteria of validity of evaluation—not objectivity but fairness. One evaluator can't make a final decision about what is the one and only objective truth independent of interpretations of the stakeholders. If the points of view of the stakeholders differ, the evaluator has to present the differences fairly.  She also has to present as completely as possible all the ascertained positive and negative aspects of the evaluated program.

 

When evaluating one program at the time this approach is appropriate. However, when building an evaluation system there should be a way to make a comparison between similar programs. The aforementioned holistic approach to evaluation is impractical in this situation, since it accentuates differences. To make an evaluation, which is useful as a part of the evaluation system, there has to be a tradeoff between the completeness of an evaluation and comparability between evaluations. The problem is that in every concrete evaluation we have to deal not only with a unique mixture of facts, factors, actions, and events, but also with unique systems of values, which are induced by interaction of the interests of the stakeholders and power relations in the social environment in question. Because of this, evaluation criteria have to be reconsidered and remodeled for every single evaluation. However, in data collection and data processing we have to use common data and procedures, which are more or less reproducible in all perceivable situations. The results of the evaluations have to be mutually comparable, comprehensive, and understandable.

 

To fulfill both of the requirements as much as possible, we propose a system of evaluation composed of two levels. On the first level, the programs should perform periodic self-evaluations based on a common model of evaluation. This would enable the programs to assess their strong and weak points and to make improvements if needed. The data collected by self-evaluations would also make it possible to monitor the quality of the programs from the outside, and to make decisions on what sort of programs are needed. This function would be supported by second level, which would consist of periodic external evaluations based on the same model.

 

Evaluation and Policy Analysis

 

In an evaluation system the issue of stakeholders and their influence on the evaluation becomes critical. As Patton (1997, 20) points out, there are many potential stakeholders, and numerous potential uses in every evaluation. To make an evaluation utilization-focused, a shift has to be made from general and abstract (potentially interested public and potential users of evaluation) to  real and specific (actual intended users of evaluation). Instead of assuming a role of impartial judge, an evaluator has to encourage deliberation and decision-making of the intended users of the evaluation. Therefore evaluation is inevitably connected to decision-making.

 

Evaluation doesn't float in some social vacuum. It is meaningful only if it is connected to other steps of preparing and implementing decisions. Its role in the fields of social security, social welfare and social work is similar to its role in the policy analysis. As Dunn (1994, 15-19) shows, evaluation gets its meaning as a part of a cycle of the problem-centered policy making process, within which policy analysis seeks to create and communicate policy-relevant knowledge about policy problems policy futures, policy actions, policy outcomes and policy performance. This is done through the basic policy-analytic procedures of problem structuring, forecasting, recommendation, monitoring and evaluation, as shown in Figure 1.

 

Figure 1: Problem-centered policy analysis (Dunn 1994, 15)

 



 These procedures follow phases of a policy-making process: agenda setting, policy formulation, policy adoption, policy implementation and policy assessment (Dunn, 1994, 16).

 

Evaluation has several characteristics that distinguish it from other policy analytic processes:

  • Value focus. Evaluation, as contrasted with monitoring, focuses on judgments regarding the desirability or value or merit of policies and programs.

  • Fact-value interdependence. Evaluation claims depend as much on “facts” as they do on “values”. To claim that a certain program has attained a high level of performance requires that its outcomes are valuable to some individual or group, as well as that the outcomes are actually a consequence of the activities and procedures which are a part of the program.

  • Present and past orientation. In contrast to recommendation, evaluation claims are oriented towards present and past outcomes, rather than future ones.

  • Value duality. The values underlying evaluative claims have a dual quality, since they may be regarded as valuable in themselves as well as desirable because they lead to some other (higher) goal. (Dunn 1994, 404)

As Dunn (1994, 404-406) explains, evaluation provides (or at least it should) reliable and valid information about policy performance; it contributes to clarification and critique of values that underlie the selection of goals and objectives; it also may contribute to the redefinition of the program, restructuring of policy problems or forming of recommendations.

 

Until now, the majority of evaluations in Slovenia assumed the approach that Dunn calls pseudo-evaluation.  They were limited to description of the evaluated programs and their effects, but avoided the comprehensive evaluation of the evaluated program. As an argument for this approach the evaluators claimed that the evaluation is a task for the agencies that ordered the evaluation. However this position gives these agencies the right of final judgment regardless of other stakeholders. Evaluation therefore can't be complete until it makes a public assessment of merit, value or importance of the evaluated program, which could lead to eventual improvements of the program.

 

There are two main sources of uncertainty and threats to reliability of the evaluation in the field of human services, especially in social welfare, which should be taken into account. The first is the ambiguity of the evaluation criteria and the second the difference between the points of view of the evaluators, especially regarding the opinion of which elements of the criteria are more important then the others. To neutralize these two threats the evaluators should discuss their individual judgments between themselves and every evaluator should explain her judgments. The common evaluation of the program should be constructed on this discussion which would bear enough consensus for all evaluators to support it.  If the evaluators are competent representatives of all the stakeholders, this procedure should increase inter-subjective validity of the evaluation and achieve the highest possible level of objectivity in absence of hard objective criteria. This way of evaluation also promotes a dialog between the evaluators, users, staff, and other stakeholders, within which the reasons for the perceived state of affairs and possible improvements can be found.

 

Obviously any individual judgment is inevitably more or less subjective. It depends on values and point of view of an individual evaluator to say nothing about other possible influences. Therefore it is important to assure that an evaluation is as objective as possible.  This can be achieved by getting several evaluators to judge the important aspects of the program independently. For the stability of results it is important to have as many evaluations as possible. Of course apart from the professional evaluators the representatives of users, staff, and other stakeholders should also take part in the evaluation. The judgments from different evaluators would then be compared.  The agreements between the evaluators could be taken for a good approximation of the objective evaluation of some aspect of the program, and the disagreements would count as indicators of uncertainty of the subjective judgments and should be discussed.

 

A second measure for achieving the objectivity of evaluation is binding the evaluation to concrete differences between the programs. In every evaluation two programs would be evaluated simultaneously and the judgments about the two would be compared. This would make comparability of the evaluations possible because the evaluations would be based on the comparison of the two programs. At the same time this would open a possibility of interchange of experiences, problems, and solutions regarding the evaluation as well as their operation.

 

If we return to stakeholders, we can broadly identify three groups of stakeholders in social welfare programs. On one side there are the users or customers of social welfare programs (and eventually their relatives), who have in common a disability to solve their problems without help from others. On the other side there are the (professional and voluntary) workers in the programs who have a double interest of effectiveness of the program and good working conditions. The state as the third stakeholder provides funds for the programs, and wants them to be efficient, effective and to cost as little as possible. Their interests partially coincide, but sometimes they can be in conflict. Therefore it can never be stressed enough that the participation of all the stakeholders in the setting of the criteria of evaluation is vital.

 

Typology of Social Welfare Programs

 

In Slovenia there is a variety of social welfare programs with different goals, using different methods, and targeting different social problems and different social groups. To put in place a comprehensive system of evaluation of so different programs, some way of classifying them should be found. The classification should make it possible to compare the programs. An obvious way of classification of the program is classifying them by social problems and target groups. However the problem with such a classification is that the programs for some target group typically have different goals and use different methods to achieve them. By trying to achieve a comparative evaluation of these programs we end up trying to compare the incomparable. It is better to construct a classification on the goals of the programs and the methods they use to achieve them. This would enable evaluators to use the same criteria for evaluation of the group of programs and compare their level of achievement.

 

Some reflection on this problem led us to conclusion that a typology of social welfare programs for evaluation purposes would have to take into account their goals and methods of work. Regarding these, a typology can be developed along two dimensions: duration of the program, and orientation or aim of the program ranging from prevention to correction. Figure 2 shows a space of this typology.

 

Figure 2: Two-dimensional typology of the social welfare programs regarding duration and orientation to prevention/correction:

 

Short term

(transitive programs)

 

 

Prevention

(preventive programs)

Correction

(therapeutic programs)

Continuous

(maintenance programs)

 

In principle every social welfare program can be positioned within this space according to its duration and orientation of its goals toward prevention or correction of the situation. According to their duration, the programs can be grouped into short term programs or short intervention services on one side, continuous programs with no time limits on the other, and between them programs of various, but limited duration. According to goal orientation the programs can be grouped into purely preventive (for instance informative) programs on one side,  mostly corrective (therapeutic) programs on the other, and programs with different mixtures and combinations of preventive and curative goals in between.

 

The described typology can be important in the light of evaluation planning, since some evaluation criteria are more and some less important for a certain goal orientation of the programs. For instance, trying to find the therapeutic effects of evaluating “pure preventive” programs is not very useful. Duration of the programs on the other hand can affect data collection strategies. For instance, while for the programs with certain duration data can be gathered at the beginning and at the end of the program, this is inapplicable for the programs of short intervention. So planning and implementation of evaluation must reflect the position of the evaluated program within the typology.

 

Criteria of Evaluation

 

The criteria we intend to use in the system of evaluation can be divided into general criteria, program type specific criteria, and criteria unique to the evaluated program. The general criteria for evaluation of social welfare programs can be deduced from the National Program of Social Security in Slovenia until 2005. In the National Program five general goals are proclaimed: improving the quality of life, providing active forms of social security, development of professional networks of help, establishment and development of plurality of services, and creating of new approaches to contain social distress. From the first three the criteria for evaluation of social welfare programs can be derived.

 

The general goal of “improving quality of life”, can be derived from the tasks stated in the National Program: “To create conditions, which would assure social security and human dignity to everyone; conditions which would contribute to social justice, to development of solidarity and acknowledgement of diversity; and conditions which would provide an adequate social status to the entire population, and prevent poverty and social exclusion.” It can be assessed by some scales that measure of quality of life.

 

The goal of “providing active forms of social security”, which is defined by the task: “To provide individuals and families, who are not in position to provide for themselves an adequate social status (through labor, social insurance, system, income from property or other any way), not only passive social care, but to provide them with support for finding a solution for their crisis, and develop the basis for putting a principle of accountability for oneself in life,” can be described through concepts of strength perspective.

 

The goal of “development of professional social networks of help”, is defined in the National Program like this: “To provide professional support for prevention and solving of social distress for all, who find themselves in situations at risk; and to assure those, who are for different reasons incapable for independent life and labor, are unable to function in their social environment, influence on implementation of programs and services, and offer them an opportunity to chose the forms of help that suit them most,” can be described with a combination of concepts of normalization and strength perspective.

 

The other two main goals of the National Program—establishment and development of plurality of services, and creating of new approaches to contain social distress—can be evaluated on the level of coverage of the space of the proposed typology. More support should be considered for the services that operate in the areas, which are covered less well.

 

Program-type-specific criteria will be defined for each type of the programs in accordance with professional knowledge about the social problems which are addressed by the programs. Two groups of these criteria include the ability of the users to function, and anticipated therapeutic effects. Within the proposed evaluation system, room is also made for unique criteria, which would be proposed as useful for the evaluation by the stakeholders of the evaluated programs.

 

Let's take a look at the concepts which underlie the proposed evaluation criteria.

 

Quality of life is a concept by which states and circumstances are described, which determine the user’s life. There are many definitions of quality of life, but for the field of social welfare, definitions from social sciences and medicine are most relevant. (Rode 2001, 6)

 

Social sciences are mostly interested in quality of life on the level of society as a whole and of specific groups strata or categories within it; quality of life is equated to welfare of the population and their satisfaction with it, and it's treated as a complement to standard of living, measured on the aggregate level. Important changes in research of quality of life were made by Scandinavian researches. On the level of scientific disciplines, they shifted the focus from economy to sociology; on the conceptual level they shifted the focus from basic human needs to control of resources of the individual for satisfying these needs; on the methodological level they shifted the focus from objective indicators measured at the aggregate level to subjective indicators measured at the individual level (Rode 2001, 47-48).

 

In medicine, quality of life is considered in connection with chronic, long-term or dying patients, where mere medical treatment is not enough (Edlund, Tancredi in Rode 200, 48).  With physically handicapped, mental illnesses, illicit drug dependencies and some other illnesses, often the negative attitude of the patients’ environment is bigger problem than illness itself.

 

In principle, the social welfare programs should enable their users to improve their quality of life at least in some aspects and the users should be able to observe this. If this is so, a change in their level of satisfaction with their quality of life between the beginning of the program and the end of the program or the time of evaluation would be an acceptable measure of the impact of the evaluated program. Of course with less intensive (i.e. low threshold) programs this impact can be less clear, so other evaluation criteria would be more important.

 

Strength perspective is a relatively new concept and, at least in Slovenia, it is not yet recognized enough in practice. It can be stated that the use of this approach makes it possible to construct solutions in cases, where classical therapeutic methods break down. It also makes everyday work of the professionals less stressful and more meaningful, and contributes to wellbeing and satisfaction of the program users.

 

According to Saleebey, “...the idea of building on clients' strengths has achieved the status of adage in the lore of professional social work.” (Saleebey 1997, 3) Everybody acknowledges the importance of this principle. However, as Saleebey stresses, practicing from a strength orientation means that everything a social worker does is predicated, in some way, on helping to discover and embellish, explore and exploit client's strengths and resources in the service of assisting them to achieve their goals. This is a collaborative approach, which relies heavily on the creativity, courage and common sense, of both clients and professionals.

 

Another term closely related to strength perspective is empowerment. Empowerment is a systematic answer to three questions in users' lives: what will I do, where will I live, and what support will I get. It is essential for the strength perspective and for the success of empowerment that a professional believes in users’ abilities and strengths. If a problem can't be eliminated, then a solution has to be built around it. In this case the user and her environment have to and usually are able to learn to live with it.  As Saleebey (1997) points out, in the prevailing defect-oriented practice professionals usually use a convex lens to look at the problem, which enlarges it, makes it closer, and at the same time isolates it. Saleebey suggests that we should use concave lens instead, which would show the problem within its context. This would enable us to see not only the problem, but also the resources, abilities, and strengths of the user and her environment, her (recent) past, and all factors, which help the user to survive in spite the problem.

 

If we use a medical dictionary for a moment, it is the human being who has to be treated, not the disease. The problem has to be pushed out of the central position and put in the place where it belongs, which is usually the place of the episode, which can be alleviated, diminished, maybe even eliminated or prevented. The main task should be to establish the life of the user as she wants it. Here individualization of treatment is of course important. However, as long as the user and the professional are focused primarily on the problem, and not on user’s survival, and resources and strength that help her to live with and through the problem, individualization often isn't enough.

 

The final goal of the strength-oriented approach is a user who is able to make independent and competent decisions about her life. The strength perspective demands a full communication which should lead to holistic healing and resilience, which is in fact a two-way process between the user and the professional. The result is empowerment, which, according to the strength perspective, is a product of work of the user upon herself and finding her own resources, strengths and abilities. The professional primarily supports this process and, if need arises, steers it.

 

Normalization is a concept, which was at first connected mainly to treating people with learning disabilities. Nevertheless it is useful in every situation where people are less privileged and marginalized because of their problems. The aim of normalization is to destigmatize the user, and empower her to do things that other people do. The user must be perceived as and average and ordinary (normal) person. In contrast with strength-oriented approach, moralization is a process initiated by a professional and it is focused on changing the environment of the user.

 

Flaker (Flaker in Rode 2001, 39-40) suggests that normalization should be a vehicle for two processes, which should enrich the society: society would get in contact with the deviant experience, and establish a dialog with it as one of the legitimate alternative of articulation of certain topics; and the critical  examination of normality and what constitutes it. The theory of normalization puts on stage the user of the services of social welfare as the central figure: if we simplify a bit, everything that helps the user to live a normal life is good. In connection with this, Brandon (Brandon in Rode 2001, 40) stresses five principles of normalization: good interpersonal relations, real choice, development of cooperation and participation, personal development—individualized development, and  integration in social environment.

 

Functional abilities of the users can play two roles in evaluation of programs. On one hand, they can limit the scope of possible effects of the program; on the other, they can be regarded as part of the empowerment process, as users’ functionality can be improved through the program. A professional (therapist) has a task to improve the user's functional abilities, to teach her to live a normal life in spite of her problems, i.e., to recognize first symptoms of the crisis, where to find help, etc.

 

In evaluation of programs we have to take into account that the level of functional abilities of users, who are included in the program, may have a big impact to its effectiveness and efficiency. Many of the possible evaluation criteria depend on the type of the users and the threshold for their inclusion in program. 

 

Therapeutic effects are changes, which are predicted and induced by a program. Therapeutic effects have to be determined by profession and by a program. The task of the evaluator is to assess whether they have been reached or not.     

 

Let us point out that concepts of therapeutic effects and of strength orientation approach are complementary, because a concept of therapy is more or less based on a deficit perspective, which is opposite to strength perspective.

 

Model of Evaluation

 

The ground level of the system of evaluation would be a level of self-evaluation of the programs which would be performed according to the same model. The cost-procedure – process-outcome analysis (CPPOA model) is proposed  as a basic model for the self-evaluation. This model makes it possible for the programs to present their work, evaluate their cost-effectiveness, and to point out the areas of operation where improvements can be made. In this way self-evaluation assumes a function of presentation and of a tool for improvement, instead of control and inspection. The model is presented in figure 3.


Figure 3:  Cost-procedure – process-outcome analysis model

 


To make an evaluation in line with CPPOA model, next steps have to be made:

 

1. Collecting and analysis of costs: most of the information on costs is collected routinely for accounting purposes. The evaluator’s task is to assure the data are complete and to rearrange them to reflect the purpose of the costs in a program. The costs can be roughly divided into three groups:  costs of the users (living expenses) and of direct work with them, costs indirectly connected to users (monitoring, administration, etc.), and other costs for operation of the program and organization. Analysis of costs should reveal the costs per user, ideally individual costs for every user.

 

2. Collecting data about users: users' personal data are in most cases available from different application forms and other documentation of the users. Evaluator should collect the data relevant for evaluation in one place. Also the data about the users' satisfaction with their quality of life and other data for evaluation should be collected from users directly.

 

3. Assessing cost-effectiveness of the procedures and processes: different indicators of improvement of users' state are used in different programs. These should be compared with costs.

 

4. Estimating a ratio of cost and benefits of the program for the community: estimates can be made about the impact of the program on costs of the problem for the community, for instance less criminality, smaller health costs etc.

 

5. Ways to improve program: based on comparison of cost-effectiveness of procedures within the program and between similar programs recommendations can be made about the merit of the program and suggestions on its improvement.

 

Resources

 

For the purpose of the evaluation the costs of all the resources used in the program should be assessed. These are all the material resources and the staff needed to run the program. Evaluation of resources can tell us what resources have been used by the program. Efficient programs will get comparably better outcomes with the same resources or use comparably less resources for the same outcomes.

 

Procedures

 

Procedures are all activities and services that contribute directly to achieving goals of the program. In the evaluated program all the used procedures should be listed and their intended effects described. The aim of the evaluation of procedures is mainly to assess if appropriate procedures are used for the established goals of the program. An efficient program will invest more energy in performing procedures which are more appropriate to achieving intended outcomes.

 

Processes

 

It's a well known fact that the same procedures can evoke different reactions with different users, or have different effects on them. Therefore apart from the procedures the intended psychosocial and other processes triggered by each procedure should be listed. The Processes are all changes, events or reactions of the users, which influence the outcomes of the program. They can improve the outcomes or work against the intentions of the program. Another way to classify the processes is on intended processes, which the program encourages, and non-intended processes, which most of the time can't be influenced by the program. An effective program will encourage the positive processes, and diminish, if possible prevent, or at least alleviate the consequences of negative processes, thus improving the outcomes. 

 

Outcomes

 

Outcomes of the program can be different psychological or social states of the users, which are altered by the program, abilities, or knowledge for solving their problems, which users acquire through the program, or other changes that are induced by the program and are in line with its goals. We can divide the outcomes into short term or intermediate outcomes and long term or final outcomes of the program.  Intermediate outcomes can be further divided into outcomes, which can signal a progress within the program and outcomes, which are used as criteria for ending a program. Long-term outcomes can be a continuation and strengthening of the short-term outcomes, or they can be further psychical or social consequences of the short term outcomes.  The outcomes can among others be evaluated by estimating reduction of total cost for community if the program wasn't in place.

 

The outcomes of the program are indicators of completion of the goals of the program. As long as the outcomes are in line with the ones intended by the program, it can be evaluated as efficient.

 

Links Between the Resources, Procedures, Processes and Outcomes

 

As it was pointed out before, every program uses different amounts of different resources, which it invests in performing specified procedures. The procedures are supposed to trigger, support, suppress, or prevent certain processes in users and their environment. Some of the processes can be predicted, some not; some can be steered, some not; some can be consequences of the program, some of a combination of other factors.  All processes together impact the outcomes and more or less determine them. These linkages have to be determined. Also the proportions of resources used in procedures, the strength of impact of procedures on processes, and the likelihood of the processes to produce the outcomes have to be estimated. These estimates can then be used for a cost-effectiveness analysis of the program and for decisions on how to improve the performance of the program.

 

Scenario of Evaluation

 

The simplest scenario of evaluation would involve two comparable programs (programs working with the same target group of users, with comparable goals and using comparable methods). Two professional evaluators would be in charge for one program. Together with a representative of the program they would collect data CPPOA about the program, organize data collection on satisfaction of users about their quality of life, normalization, assess the level of strengths orientation in the program, and collect information about other evaluation criteria. The evaluator and the representative of the program would prepare a presentation of the program based on the collected data.

 

The four, two evaluators and their two coworkers within the programs, would then individually rate the aspects of the programs relevant for the evaluation on basis of the presentations of the programs and their experience with them. This way we get a complete design of factors that can influence the ratings: insider/outsider point of view, experience/no experience with the program, professional/evaluator. The four sets of ratings and comparisons of the programs based on them would then be discussed within the evaluation group, and if possible a common rating would be produced. These ratings and the presentations of the programs would then be presented to both of the programs and other stakeholders, and a discussion on the results of the evaluation organized.

 

Implementation of the System of Evaluation of Social Security Programs in Slovenia

 

The proposed system of evaluation is supposed to be implemented gradually and in the process altered to suit real needs and wants for evaluation. At the end of the process we hope to have a system of regular evaluations of social security and social welfare programs in Slovenia. Special care will be taken to make the additional burden of evaluation to the staff as small as possible while assuring the regularity of collection of the data needed for evaluation.

 

For valid and reliable evaluation, which could assure the improvement of the programs, data should be gathered about the resources used, procedures implemented, processes detected and outcomes reached, on the individual level for every user, and aggregated later for the program. Collecting data directly on the aggregate level of the program diminishes validity and reliability of evaluation. The data aggregated on the level of the program can still be sufficient for a minimally relevant evaluation of the program. However, the comparison between the programs based on data collected this way is not necessarily valid, since they don't allow us to locate factors that are interrelated within the programs, and which influence the outcomes.

 

Nevertheless we presume that in the first stage of forming the system of evaluation collecting data on the individual level would be too much for the staff of the evaluated programs to handle, since the proposed model of evaluation demands a different approach to monitoring of the events in the program. This needs a whole different approach from the one prevailing in the social security programs at this moment and could interfere too much with the daily routines within the programs. For this reason we settle for data collection on the aggregate level of programs for the period of introduction of the proposed model of evaluation. In the second stage, when the evaluation system is more or less established and structures of CPPOA models are stabilized for different types of programs, we presume the need will arise for more detailed data. Data collection on the individual level, which will allow better decision-making, will then be introduced.

 

 

 

References

P.P. Merino (1999), Treatment – Evaluation Literature. V: E.M.C.D.D.A, Evaluating the Treatment of Drug Abuse in the European Union. Luxembourg: European Centre for Drugs and Drug Addiction (25-28).

 

B. Mesec et al. (1998),  Evalvacija preventivnih programov centrov za socialno delo 1995-1998. Ljubljana: VŠSD.

 

U. Nizzoli (1999), Treatment Information Systems: The Emilia-Romagna Experience. V: E.M.C.D.D.A, Evaluating the Treatment of Drug Abuse in the European Union. Luxembourg: European Centre for Drugs and Drug Addiction (63-66).

 

NPSV (2000), Nacionalni program socialnega varstva do leta 2005. Uradnilist Republike Slovenije, 10, 31: 3777 – 3787.

 

M.Q. Patton (1997), Utilization-focused evaluation. London: Sage Publications.

 

N. Rode (2001), Mo?nost evalvacije nevladnih neprofitnih organizacij na podro?ju socialnega varstva. Univerza v Ljubljani: Fakulteta za dru?bene vede (magistrsko delo).

 

N. Rode, L. Rihter, J. Zorn, B. Kobal (2003): Oblikovanje sistema evalviranja izvajanja programov socialnega varstva: poro?ilo za leto 2003. Ljubljana: Fakulteta za socialno delo.

 

D. Saleebey (ur.) (1997),  The strengths perspective in social work practice. New York: Longman.

 

L. Šugman Bohinc (2000), Kibernetika spremembe in stabilnosti. Socialno delo, 39, 2: 93-108.

 

W. Wolfensberger (1983), PASSING Normalization Criteria and Ratings Manual. Downsview: National Institute on Mental Retardation.

 

2000, Nacionalni program socialnega varstva do leta 2005, Ur.l. RS. 31/2000 (str.: 3777 – 3787)

 

2002, Na?rt za izvajanje usmeritev in nalog iz nacionalnega programa socialnega varstva do leta 2005, Ur.l.RS 45/2002

 

ASA, 1998, The Program Evaluation Standards, Summary of the Standards            http://www.eval.org/EvaluationDocuments/progeval.html 11. oktober 2004

 

Novak, Mojca, (ur.) 2000, Enotni informacijski sistem socialnega varstva v Republiki Sloveniji, zbirka Trenutek. Inštitut RS za socialno varstvo, Ljubljana

 

Patton Michael Q., 1997, Utilisation-focused evaluation, Sage Publications, London,

 

NPSV (2000), Nacionalni program socialnega varstva do leta 2005. Uradnilist       Republike Slovenije, 10, 31: 3777 – 3787.

 

N. Rode, L. Rihter, J. Zorn, B. Kobal (2003): Oblikovanje sistema evalviranja         izvajanja programov socialnega varstva: poro?ilo za leto 2003. Ljubljana:     Fakulteta za socialno delo.

 

N. Rode, L. Rihter, J. Zorn, B. Kobal (2004): Oblikovanje sistema evalviranja         izvajanja programov socialnega varstva: prvo delno poro?ilo za leto 2004.    Ljubljana: Fakulteta za socialno delo.

 

B. T. Yates (1999): Measuring and Improving Costs, Cost-Effectiveness, and Cost-Benefit for Substance Abuse Treatment Programs: A Manual. National Institute on Drug Abuse, Maryland; US Department of health and Human Srevices: National Institues of Health. Available at:             http://www.drugabuse.gov/IMPCOST/IMPCOSTIndex.html

 

Back to Top

Copyright for the I.U.C. Journal of Social Work Theory and Practice is owned by the Social Work Program, Department of Social Relations and Services, Bemidji State University, Bemidji, Minnesota, USA. One copy may be made (printed) for personal use; teachers may make multiple copies for student use if the copies are made available to students without charge. Permission must be secured from the editors for sale of any copies of articles or for any commercial use of the material published in the Journal.
2001 Copyright BSU/IUC Journal of Social Work Theory & Practice