Process for running short surveys

This document defines the process for running short surveys on MDN.

In 2019–2020 the MDN team ran the MDN Web Developer Needs Assessment (Web DNA), the results of which we made available on the MDN Insights site. This has produced a tremendously useful set of results that has actively shaped the work of browser vendors and other orgs involved in standards over this time.

However, running a Web DNA requires a huge amount of effort and resources across a number of different companies. We wanted to provide a mechanism by which we can run smaller, more focused surveys that produce more easily actionable data. These would help us find directions towards solutions to specific problems.

Benefits:

• Much quicker to take for developers, therefore causing less survey fatigue.
• Good to get a quick indication on certain topics.
• Quick turnaround time for stakeholders on getting results.
• Can be run with a good cadence, e.g. monthly.
• Can be exposed for any percentage of MDN visitors. E.g. key ones can be exposed to 50% of the visitors, regular ones to 5% of them.

If you want to submit a survey proposal, you can do so by filling out the short survey proposal form. This will be reviewed according to the survey process.

To be suitable, the survey proposal needs to be:

• Of interest to the MDN audience.
• A topic which the stakeholders would be willing to address/follow-up on. E.g. will the results inform improvements to the web platform or MDN?.
• As concrete and detailed as possible, avoiding high-level questions which are hard to make actionable and addressable.
• Not more than 3-5 questions, and the survey should take no longer than 10 minutes to fill out.
• Belonging to a clear category, e.g. CSS, JavaScript, Accessibility, documentation.
• Not asking for any personal identifying information (PII) which isn't included in the privacy policy for the survey takers (or asking for PII that is inappropriate for the agreement provided).

1. Submitter submits a proposal through the submissions form, outlining:
   • Topic.
   • Category, e.g. CSS, JavaScript, Accessibility, documentation...
   • Objective for the survey.
   • Date (range) when the survey ideally should be run.
   • Actions which will be taken based on the results.
   • Questions and answer options

     Note: Certain types of data will need a legal review, along with intention, e.g. is the data intended to be published, or shared with other organizations? See Legal requirements below for more information.

   • Suggested percentage of the MDN to be exposed to the survey, e.g. 3%. The Default percentage is 5%; if you want a higher percentage, a clear justification is needed. For example, if your survey aims to collect data to help inform solutions for a well-documented critical web platform bug that is affecting a large number of developers right now, then that is probably good justification, as long as you can provide some clear evidence. Typical criteria to base this justification on can be found at OWD prioritization criteria — these criteria were written to inform prioritization of MDN content, but they are still relevant here.
   • Suggested parts of MDN to show it on (e.g. CSS pages).
2. The review team reviews submissions within a week, and:
   • Asks for more information if needed.
   • Approves/denies the proposal. The MDN content team holds final veto power for which surveys will be run and how.
   • Applies a priority level to the proposal, e.g. P0, P1. This is a rough guide used to stack rank the proposals.
   • The review team will consist of 3 members of the MDN Product Advisory Board, chosen on a case-by-case basis.
3. The review team creates and agrees a final survey in partnership with the submitter, based on their initial questions.
4. The survey team tests the survey thoroughly on desktop, Android, and iOS to make sure that it does not contain any layout/rendering errors that would lead to frustration with filling it out.
5. The review team slots the short survey for an agreed date and duration.
6. The review team provides the results and data to the person requesting the survey and any other relevant parties.
7. MDN publicly presents the results of the survey on insights.developer.mozilla.org.

Mozilla cares deeply about user privacy, therefore any survey we run needs to be very carefully considered in terms of what data we collect, how long we keep it for, and how it shared and published. Mozilla legal will often want to review a survey proposal to check that it meets with our requirements.

• Generally, we don't need legal review for most information you might want to collect. This includes information on user's personal preferences and opinions, choice of browser, job title, etc.
• However, if you are collecting any personal identifying information (PII), sharing information, or publishing information, we will need a legal review. This includes name, age, postal address, email address, sexual orientation, religion, or anything else that could be used to identify or profile someone.

A good guideline is to think about the minimum amount of data you need to get useful results, and not collect anything more than you need.

Also think of answers to these questions:

• Who will need access to the data? Are you intending to share it with any third parties? If so, who?
• Are you planning to publish the data? If so, where, and in what form? The raw data, aggregated data, both?
• How long are you intending to keep the data for? Unless you explicitly express the need to keep it for longer, the data will be deleted after 6 months.
• Are you intending to use any kind of alternative tool for the data collection? Mozilla tends to do everything using our Survey Gizmo/ Alchemer account.
• Are you intending to join any of the survey data with telemetry?

The MDN team at Mozilla owns the survey process, are responsible for publishing the survey, and holds final veto power for which surveys will be run and how. We will make sure that the appropriate legal notices and privacy statements are included on the header page of each survey.