The following are the outputs of the captioning taken during an IGF intervention. Although it is largely accurate, in some cases it may be incomplete or inaccurate due to inaudible passages or transcription errors. It is posted as an aid, but should not be treated as an authoritative record.
>> MODERATOR: Who's here? OK. We'll start with working group 2 and then come back to Nicholas.
Part of the team that made the report and going to report on the future of working group 2, which is on education and skills. The reason that topic is there, is there's a huge gap between the skills that are offered by tertiary cybersecurity education and in general, the society and industry makes.
We'll present the report formally tomorrow. We'll not really go into that here. Do that tomorrow, with official handouts to the chair.
Going to present on what we're going to do with the outcomes and the ideas of the report. And what our future for 2023 is. Thank you.
>> Hello? Yeah. Thanks for the introduction without. I'm from the Netherlands and joined the IS3C not that long ago. I helped on the research on the gap between the skills of tertiary graduates and what people in the industry expect.
And yeah. As Wout said, the official report will be presented tomorrow. Not going to talk about that yet, but I am going to talk about the future and the goals for 2023 are as follows.
First of all, we will refine and pilot the competence model with partners in several different countries and regions of the world. To gather input and insight into means of implementation.
The second goal is that we plan to work with experts from diverse backgrounds and sectors to begin developing an education kit that can be used in educational institutions and also by individuals in the context of life‑long learning.
Third is we will continue collaboration with education and industry sectors to find ways to improve knowledge‑sharing and collective good practices.
We also plan to look into means of scaling up some of the good practices that have been identified in the study.
And finally, we are very eager to work at industry and education leaders to look at ways careers in the cybersecurity sector could be more appealing to young people and especially girls. That's our plans for the next year.
>> WOUT DE NATRIS: Thank you, and that's something we're going to look for support for and contribution for.
Because the (?) we want to send up will take a lot of effort from around the world to work together in. So the next step is that we are able to announce two working groups to start in 2022. We have three actually at this moment. The first one is on the Internet of Things security by design. Second is the education skills one. And the third is on data governance and security.
The third one ‑‑ fourth one, at least we are working through three, sorry. Because we have devised that already over two years ago but finally found the funding to start it. And Mallory Knodel has been waiting all these years to actually start the work.
But she's going to present on what our plans are for the very near future and what sort of outcomes that we are going to present at the next IGF. Mallory?
>> MALLORY KNODEL: So, happy to, as Wout said. Just checking my notes here. So the point of the third working group is very much I would describe it now, if you were in the room for the previous session, trying to actually analyze and understand what are some of the barriers then, to procuring products, software, hardware, that has implemented the proper standards.
And so I think there's a lot more to it than that. So of course, one of the first steps of this work is to research what's already been done. What are groups already focused on. Knowing that there may be procurement policies and supply chain management considerations in many different governments already.
In industries. And so on. So trying to scope and map the current landscape. It's a pretty typical first step. So we'll be asking for interested researchers to help us with that first issue paper. I think what's important about also the methodology we've devised or the approach we're taking, is that because this is a dynamic coalition, we want to bring folks along with us.
So if we are discovering or citing publications, we want those authors to know about our work so they could potentially get involved. If we have researchers coming along, we're doing qualitative interviews. We want the interviewees and the researchers to stay engaged in this dynamic coalition for the long‑term. So that's one of the other sort of objectives of this work is to not just produce knowledge, but also create community around this issue.
Another thing to note about the way we've devised the work is that we want to make this as specific to the IGF's mandate as possible. So recognizing the UN and the IGF in particular has a certain sphere of influence. Has a certain perspective that could be different than one single government or even regions or industry.
We want to make sure that our issues and our guidance are aligned with what the IGF is scoped to do. And not be duplicative, but as synergistic as possible. Once we've scoped and mapped and analyzed, I think the point will be to issue guidance. I think we should take the time we need on that first step, to ensure that the guidance we produce is actually actionable and relevant.
So we shouldn't, I think, get too much in a hurry to do that guidance work. But of course, then once we have guidance out, it can always change. And it's not the end of the conversation. In many cases, it might be the beginning for folks who are just going to be getting involved and will find that guidance actually useful.
So that's hopefully a pretty straightforward explanation of the approach we're going to take. And so for interested folks in that work, we can obviously take more activity‑based or step by step approach to that. But I'm envisioning this to be very much a group research exercise.
And so, yeah. Hoping for participation from the community.
>> WOUT DE NATRIS: Thank you, Mallory. And what my personal opinion about the work that we are doing is, is that this topic is probably the most important one that we will do, without disqualifying all the others, of course. Because they all do important work.
But the fact is that if governments and large organizations are going to adhere to the principles that we're identifying around the world, then that will mean it becomes a driver for industry to actually deploy these modern standards.
If you're going through a procurement process where the standards are not adhered to by the company applying for the procurement, then would not get the assignment. So that should be the basis of every procurement process in the near future. To make sure that you buy secure by design.
And whether it's for IoT, for e‑mail, for transport on the Internet, or for software, that will become a driver to be more secure.
The next working group we'll be announcing that starts work in 2023 is going to be led by myself. And we call that, "the list" basically, a bit unofficially. The idea behind that is that if you are a government anywhere around the world and you are confronted with all the standards that are out there all that once, probably you will not even start thinking about them, because there are too many of them.
So if we can identify, and that is going to be what we are going to try to do, identify a consensus on, let's say, the top 40 most important standards and ICT best practices related to cybersecurity, and present that to the world to start working with, that would be a tremendous incentive to actually start working with them.
Because you can for a while ignore the others. The idea that we have to start that is to find experts around the world that are willing to sit in to three, four, five sessions, over a year. To identify not only the standards, but also the categories of standards. What we should divide them in to make sure that it takes up all the different elements of the cybersecurity.
So not just your ISP, but also your hosting, your website, your software, et cetera.
So that from all angles, there will be an adviser in cybersecurity. The experts, for example, the Internet engineering task force from the worldwide web coalition. But also the IEEE, which stands for atomic engineers, something, do you know? [Laughing] I don't know everything by heart. Also ICANN. So we have experts that come to ‑‑ diverse experts that come to some sort of consensus.
There's one example in the world that I know of that is the Dutch list containing, I think more than 46 standards. And it's aimed at governments, voluntarily. But it is called either apply or explain list. And the explain is why do you not buy secure by design, what is the reason that you have to explain that to government agency.
So that is something which is mild driver for governments to procure, and will be one of the best practices that we will be looking into.
The second list we'll try to compile is to have the complete overview. So what are we talking about when we talk about cybersecurity related Internet standards and best practices? If we can have a repository like that, which is also a live repository, it means anyone could access that list to see what is it I actually have to deploy, why am I deploying it, what is the issue, what does it solve, and where do I get the official information if I need more information on that specific standard?
So that's something that we hope to start with the standard‑making bodies and to make sure that this list is hosted somewhere where it can be live and also taken care of, basically. Those are the two things we're going to try to do in 2023 and we got the funding for to start the work on.
The first reach‑out has been made here at the IGF for persons within organizations to actually start participating.
So that is number 5. The next one that we hope to say something about is working group 7 on consumer advocacy, and that is something that we are investigating at this points with representatives of the Brazilian government and hopefully we can start that somewhere in 2023. So this is not announced, but it's in the works.
The idea behind that is when consumer organizations test products with IoT components, usually what they test is the quality of the refrigerator or the coffee machine or the telephone. Does the light work, how does it open? Does it close right? And the environmental stuff behind it.
But the ICT component is often not tested. So if that is insecure, everybody buys an insecure fridge, IoT device. So if consumer organizations get some blueprint on how to test the IT components in the products they test. That would show if the producer, if the manufacturer of these devices actually comes up with a secure device.
And you can make it literally a choice. Hey, this is an insecure device; I'm not going to buy that one; I'm going to buy that one.
And the different industries involved say, hey, I've got a green one, and the other one has red. Or the other way around. Perhaps I want to get a green one as well. And then you can see the Internet.nl with the hall of fame, saying hey, Michael has 100% and I score 15. I had better get working to score that 100% as well.
That could be another driver for industry to actually adopt these standards that are available. The next working group that we'll be presenting is on quantum computing. And we have two proposals for that.
The first proposal, the lady could not join today because of family circumstances. The other, so I'm giving the floor to Elif.
>> ELIF KIESOW CORTEZ: Is it possible to hear me?
>> WOUT DE NATRIS: Yes, I hear you. Thank you.
>> ELIF KIESOW CORTEZ: Thank you. I'm Elif Kiesow Cortez. Currently I'm based in the Netherlands. As Wout mentioned in the opening, regarding the plans for 2023, I would like to suggest that our D.C. creates a new working group focusing on the governance of two emerging fields. AI and quantum technologies. I think these technologies would fit in our mandate given their dual use nature.
We observed there's increasing attention from the tech companies in terms of investments and from the policymakers in terms of regulatory efforts, such as the AI acts both in the EU and the US.
I think research into security risk‑mapping will become even more crucial in the coming years. Therefore, we believe that having a new working group on the governance of emerging technologies would fit our coalition's mission very well.
There is also breakthrough developments in the quantum technology field at the moment. Big tech companies such as Amazon, IBM, Google and Microsoft, already reported that they have launched departments dedicated to quantum technology.
And in its March 2022 report, the U.S. national science and technology council stated in the next 1‑3 years, agencies should be established for addressing the legal and policy issues associated with quantum technology.
As this timeline and current developments demonstrate, this would be a great moment for our coalition to get involved in the governance discussions of these two domains.
I envision that the mission of this working group will be to offer a road map for anticipatory governance strategies for the field of emerging technologies, initially focusing on AI and quantum technology. The proposed governance road map will be addressing the relevant roles of the state, private sector and Civil Society stakeholders based on the lessons learned from past governance efforts concerning complex technology domains.
We observe there are several funding options in both domains. So we expect that we will be also able to attract external funding for this new working group, from institutions willing to be in a leading position in the governance of these fields. We envision the deliverables will include mapping current risks and opportunities associated with these domains. Providing policy recommendation report with input from diverse stakeholders, as well as helping with standardization guidelines, based on this policy recommendation report.
Thank you very much. I look forward to hearing your feedback.
>> WOUT DE NATRIS: Thank you, Elif. I just remembered that I received the mission statement of Narine Kachatryan. I hope I pronounced that right. From Armenia, on the same topic. So we will be investigating whether they are to be merged in the near future, or that they are truly separate.
Also on quantum computing and excuse me for reading her document.
In recent years, there has been a substantial amount of research in quantum computers, machines that exploit quantum mechanical phenomena, to solve mathematical problems that are impossible for regular computers.
When they become ‑‑ exposed to massive risk. This could be serious compromise the confidentiality and integrity of digital communications, on the Internet and elsewhere.
On the other hand, (?) is involving two and the development of quantum resistant or post‑quantum (?) helps create cryptographic systems secure against both quantum and classical and quantum computers. New key infrastructure challenges and the need to update algorithms in use.
As quantum computing emerges security standards will change. North America to Middle East and Africa and everywhere else are already prepared?
Effective management and adaptation of their PKI infrastructures and machine identities to new algorithm standards and environments. What skill gaps do they have? IOC coalition is in process of establishing this working group, nine quantum computing and post‑quantum encryption with specific aim of reviewing current quantum computing and practices around the world. Coherent package of global recommendations. Maintaining relationships with related technical community security leaders, engineers and developers and other interested organizations.
Among the far-reaching goals we propose the development of best practices on encryption, support the quantum safe cryptography community to protect data.
At this moment, this working group is based on voluntary contributions of experts and we would welcome anyone who would like to join in this work.
So that is another working group that we hope to start in 2023. Because it's not currently active. But this is the mission statement that was developed by Narine, and hopefully she will be able to join Thursday in our session then. Thank you for joining us. I understand you were leading another session just now. Nicolas Fiumarelli is the chair of our working group 1 on security by design for the Internet of Things.
We are going to present this at the dynamic coalition. Our draft report pretty soon. Which we will not go into depth here. But we will be talking about the work in 2023 that has been identified. So the mic is yours.
>> NICOLAS FIUMARELLI: Thank you thank you. Hello, everyone. Nicolas Fiumarelli here. I will share the screen rapidly for you. But we are in the networking session to talk about the future, right? So what we did this year was a report on research about the Internet, insecurity, with more than 30 policy documents around the world focused on cyber security. We were trying to do top 20 of best practices that would be recommendation for policymakers to include Internet security in national plans.
We know the Internet of Things, the devices connected to the Internet are increasing from year to year. This is a graphic that shows that.
For example, you can see in 2023, it's expected to have more than ‑‑ in 2030, expected to have more than 30,000 million devices. So there are a lot of threats. Service attacks and the combination of these devices can be used to attack a server.
And other factors, right? Like personal data, sensitive data, been shared with thieves. Malware, corruption of the firmware inside the devices.
So there are a lot of things.
This is a number of the research that take part. We have different kind of document. We start ‑‑ we didn't take into account the recommendation from the standardization entities, but we started from the policy document, the regulatory document, because we wanted to know what are the current practices being taken place in terms of requirements or recommendations. But from the policymaking platform, could a national government body, regulatory frameworks and code of practices.
So for the future, we would like to know, we have some questions for you on how to proceed with this research. We know that in 2022, there is a lot of loss on cybersecurity been working. And a way to include all these best practices in the report.
And we would like to know well, what do you think, because this is a major topic. It's an issue that we are currently facing and we expect the quantity of devices in the following years. We will be in a situation that we will need to take some actions to prepare for what the attacks will be, and how to protect our devices.
And at the end, that is the idea, right? To protect the personal and sensitive data that could be badly used.
So at the end, it's a whole issue. So would like to know from the audience, for the last part, what are your ideas for future work here at the IS3C, regarding the different working groups and specifically the working group 1 that is Internet of Things and security by design.
>> WOUT DE NATRIS: Thank you. And the final working group somebody has announced to start in the near future is on three specific standards. DNSSEC, RPKI, and (?). Deploy. And the idea behind that is to start a working group on bringing different stakeholders together so that it's discussed not from the technical angle, but from the deployment angle.
Just like Mallory says, what are the obstacles. And if we focus on three different standards, that may actually help the exercise in the future as well. And that is the idea somebody brought up, which we will be looking into pretty soon.
So that basically ends our presentation. And pleased to have the opportunity to have some interactive conversations. So who has any questions on the work that we've been doing and that you are able to ask now? So the floor is basically to you.
>> OK. Thank you very much. I appreciate your presentation. I think difficult, my name ‑‑
I work with Ministry of Science. Regarding my question or suggestion. There are few Ethiopians or a lot of Ethiopian ‑‑ in different university. We have more than 14 university in Ethiopia. Graduate with ‑‑ ICT and other ‑‑
So based on your presentation, a ‑‑ maybe need skill training regarding your presentation, like quantum digital technology. They don't have skill training.
So did you give skill training for ‑‑ (?) or do you have a plan to give scholarship for training, devices for Ethiopian new graduate students. Including me, because I have no PhD program.
So give me PhD scholarship for training device, I'll come to your country. Thank you.
>> WOUT DE NATRIS: Thank you for that question. I'll take it first and then can add on the basis of what we found. We are identifying the skill gaps in education. And what we have found is that ‑‑ let's explain first that we've interviewed people around the world and we did a survey.
And also, we had responses from 66 different countries and literally from everywhere in the world. And from Africa and the Asia Pacific, especially, we heard from people working in the ICT industry that they lack a lot of the skills necessary to work in the industry.
But what they also lacked is that they could not find the right sort of training online. There were no training courses online explaining specific topics. And I think that we will not be able to teach or give teaching programs, because we just are identifying the skill gap. We have done that. I have the report, about 15 minutes in front of me, going to present that tomorrow at 16:15.
But we are going to try and do in the future, as just explained, is to set up working committees in regions in the world to discuss the skill gap and how to close it in the future. So that means that we have to bring together different sort of stakeholders.
It's not just the schools. It's not just the government. It's not just singly the industry. I think the three of them together with academia will have to decide on how do we make our curricula, literally up to date so that a teacher is not teaching the skills of 20 years ago, but is teaching the skills of, maybe two years ago. But not 20 years ago.
And that is a program, that we may, if we find the funding, be able to set up a tool kit for and some sort of capacity training program, which we can then hand over to the regions and say OK, we've developed this. Now it's up to you to go to your national countries and make sure that it's deployed.
Because that's the extent that we in our little configuration that we hopefully can get to. If we achieve that, then everybody has a tool kit to work on these skills together and to perhaps set up these online trainings for youth around the world to keep their skills up to date. Because if we get to that level, then people would start providing the sort of knowledge online that would help the world tremendously.
But that is, I think, as far as we can go. We can't give any scholarships or whatever, because we are a very small dynamic coalition with very limited funding. But we do hope to provide the tool kit. Would you like to add?
>> I think one thing I would like to add is that we are aiming to get the education and the industry to work together better. Because if we know what the industry expects from the graduates, then it's easier to adapt the education to them. And that is what ‑‑ yeah, is missing in the gap right now. So we need the industry and the education to work together better.
>> WOUT DE NATRIS: Thank you. I'm the vice‑chair of this working group and she was supposed to be online, but apparently there is some issue there. Thankfully, you were able to step in at the last moment. So thank you for that. Is there another question?
So we can be extremely ‑‑ we have been extremely clear then, I think. So thank you for that. Is there any comment online Mark, that you would like to share?
>> Mark Carvell: I'm following the chat and tried to stimulate some discussion online with questions like, "What can we contribute to the digital global compact?", for example, as well as if people look at our priorities, we're extending into new areas on the basis of proposals we've received from individuals for the future expansion of the scope of the coalition.
So, as I said, I'm inviting people online, as well as in the room to comment on our priorities, bearing in mind, potential contribution to the global digital compact. So I put that source out to you in the hope that somebody might wish to comment, react.
Back to you, Wout.
>> WOUT DE NATRIS: Thank you. Anyone would like to comment in the context of the global digital compact? No? Then I'm going to look at the panel for the famous last words. And start with you, Mallory.
>> MALLORY KNODEL: Yeah. I guess I wanted to reflect after hearing others present and questions from the floor, on the real theory of change here. I think one of the things we ‑‑ in trying to solve this problem, and for those of you present in the session just before this. There are just many different gaps all along the path here, from creating secure standards to then end users being surrounded by products and services that actually implement those standards.
It's far more complicated than I think any one effort so far has managed to address. And maybe the point here is that we're kind of looking at all the different problem spaces.
But one of them that I know I will come up against in the work to research procurement and supply chain issues is there simply may not be enough products out there that would meet a high bar for security and safety and procurement policies.
And then, so you have some tensions with that fact. So if there aren't enough products that meet your high standard, then you have also, potentially competition issues. You might have monopoly issues if the standards are not always open.
We assume we're talking about open security standards, but sometimes they're not open. There are plenty of standards where issuing the standard is part of their business model. So I'm just thinking about in a year from now or so on, we might actually have a better idea of what are the hard problems. Because there are inherent tensions between creating products that are in a competitive market. But then we can leverage to make sure that they're ubiquitous. I'm just not sure that's always as easy as it sounds.
>> WOUT DE NATRIS: Thank you, Mallory. Yes, obviously, there are a lot of obstacles in this field, otherwise these standards would have been deployed 20 years ago, because some are around for more than 20 years. I see a question.
I'll first go around the room. First, go down the table.
>> So thank you all for being here and listening to us. As Wout said, tomorrow we will be presenting the report. And I would say the report is only the first step. We defined the gap and the next step is to, well, hopefully solve the gap and to work on that.
So if you have anything to add, or if you could contribute in any way, feel free to reach out. And we're looking for best practices, but we're also looking for problems that you need best practices for. So if you feel like anything is missing in education or if you're in the industry and you feel like something is missing in the skills of the graduates, then feel very free to reach out to us. Thank you.
>> About the global digital compact, there is a road map of the global digital compact by the Secretary General, and every company, every organization around the world can contribute to the global digital compact until March 2023. And for the IS3C work regarding the global digital compact, I identified three of the key points of the road map. One way, promote digital (?) security, this could be made by deploying all the standards that have been proved that solves some of the security issues around the world.
There are several standardizing entities that we are looking for all the report. And another one is ensuring the protection of human rights in the digital era. While deploying these standards, you mean be helping preventing these threats to happen, right? Not only about the IoT security, but for example, new emerging working groups like the RPKI, RSEN (phonetic), there are several threats. Insuring protection of these human rights.
And the last one is strengthening digital capacity building. Because there is a need for digital skills for the workers in the cybersecurity market or at different organizations to take into account all these things and to know what they need to do to protect the products, to protect their citizens.
So that will be my comment. So there is a link with three of the key points from the global digital compact.
>> WOUT DE NATRIS: Sir, your question, please.
>> I will give kind of opening question. As if ‑‑ before I was working in the Ministry of Technology in Ethiopia. So maybe this is a good opportunity to announce my own reviews.
So under the Minister of Technology, we have (?) technology department. Under that, we have many different countries based on their embassy or consular interest. So while research ‑‑ can have relative understanding with ‑‑ it's open. So more on them understanding signing with different projects, technologies.
So if you have, welcome to ‑‑ and your country researcher. So maybe we have longer term, short term, and medium‑term training, including knowledge‑sharing programs.
So you're welcome to any country from this. Thank you.
>> WOUT DE NATRIS: Thank you very much. You would like to make a comment? Yes.
>> Hello. I'm from Brazil. I'm sorry I'm a bit nervous. This is my first IGF. And this is my first question. I'm from [Laughing] youth program from CGI. I have ‑‑ I want to know how the young ones like me can engage in the activities or the working groups.
>> WOUT DE NATRIS: There are two next to you who know exactly how to do that.
I'll respond to you, sir, that also your youth researching at this moment, are very welcome to join this initiative. First step that you have to take is go to the IGF website and register, Internet coalition Internet standards and safety. Sign up for e‑mail list. That means you've become, between brackets "a member" and that means you get all the information about working groups that start, information questions, research questions that you can participate in.
So at that point in time, you can actually start saying I want to be part of working group 3 or working group 5, or whatever number, and then you become an active contributor to this process.
And with that, everybody from around the world is welcome to join this process. Because the more voices we have and the more knowledge we're able to get into this dynamic coalition, the better and more effective our programs probably are going to be.
So for both of you, please join or tell your colleagues to join. Because that way we'll have more voices to the table. And that, I'll almost have my concluding remarks. Looking to Mark, any final comments online or from yourself, Mark? Then I'll conclude the session.
>> Mark Carvell: Thank you, Wout. I don't see any new comments online. But I would like to just add one point. And I thank the gentleman from the Ministry of Technology here in Ethiopia for that contribution.
My point is that it's important for dynamic coalitions like this to be aware of potential interest of other initiatives, of other government departments, ministries, looking at this whole area in cybersecurity online.
And identifying ways that through collaboration, and engagement in IGF initiatives like this dynamic coalition, we can make progress in resolving some of those problems and some of the emerging ones that solutions have yet to be identified for.
We're in that business. That's the aim of this coalition. Our tag line is making Internet more secure and safer. So I invite participants here to let us know if they are aware of initiatives that are happening similar to ours where there could be useful synergies, in all regions. That would be very valuable information for us.
And also events, opportunities for this coalition to promote its profile and disseminate its outcomes. Like the report on skills that was described earlier. We have a report and we need to find opportunities to promote awareness of that report and to get feedback and to build on that report and undertake further work.
So thank you very much for coming today and we really appreciate your interest, and please network beyond this networking session to help us advance the work further. Thank you.
>> WOUT DE NATRIS: Thank you, Mark and thank you, panel, and also Elif online, for contributing to this networking session. I think it shows exactly how ambitious we are in trying to get this topic of deploying Internet standards across.
With that, let me just point out that we already discussed the need to have people contributing to this. Willing to do a little bit of the research to help the project leads to write the final reports, to get the right content from around the world and initiatives so that we know what initiatives are around in your countries.
So that is the thing. I think that's something you can help us with. The other thing is that we're looking for funding. We've been funded so far by several different sort of organizations. We're now going to be funded by the RIPE fund, SIDN, we've been funded by. But also Microsoft.
So we have a very diverse set of people who are vocal organizations that are sponsoring our work. That's something we're looking for in the future as well and work hard on almost every day, to make sure that the work that we devise is also going to be taken care of, basically.
So that in the IGF in Japan in 2023, we are able to make new presentations or new reports.
But also to have this ‑‑ the recommendation, the guidelines coming out of the report, recognized as an IGF output.
I think that is something which is going to be tremendously important as well. Because in the end it has to be recognized as an official outcome of IGF process. And that is something we'll be discussing this afternoon.
Without ado, I want to thank you for your contributions here, for being present, and also online. And let's hope that we can be even more successful in the future.
Because if I look where we come from from two years ago where we started, I think we have made tremendous progress with presenting two draft reports at this IGF, and hopefully next year, maybe even four.
So let us move forward. Thank you for being here and have a very good rest of the IGF and safe trip home. Bye‑bye.