Internet2 Preliminary Engineering Report

(January 1997)

1. Context and Design

This is a working document that summarizes the work thus far of the Internet2 Engineering Working Group.

An earlier document by a subset of the current group outlined an architecture for Internet2. In this document we build on the earlier group's work, describing the general requirements for the Internet2 communications infrastructure, defining the parameters and constraints that must be considered in the development of that infrastructure, and suggesting specifics of architecture that should guide subsequent detailed design and implementation.

1.A. Application Requirements

Within and across many colleges and universities, a set of advanced network-based applications is emerging. These will greatly enrich teaching, learning, collaboration and research activities. For example:

• The broad use of distance learning will require selectable quality of service and efficient "one-to-many" data transport in support of multimedia and shared information processing.
• Our leading-edge research community needs high capacity and selectable quality of service to make effective use of national laboratories, computational facilities and large data repositories.
• Medical researchers need support for remote consultation and diagnoses over highly reliable and predictable communications services.
• Physical scientists, especially those who deal with massive astronomical or geophysical datasets, have similar needs.

As transaction-level commercial data come into research focus, financial and economic analysts will need real-time access to masses of data.

And the list goes on. A major impediment to the realization of these applications is lack of advanced communications services in the current commodity Internet -- there is, therefore, not yet a testbed in which to prove their effectiveness, much less to implement them on a wide scale.

Internet2 seeks to enable these new applications. It will do so by working with the information technology industry to develop common standards and support services for new classes of applications and to ensure the availability of the advanced communications services required. Many of the communications service technologies required are still in the pre-competitive stages of development. They should benefit greatly from a broadly based testbed encompassing the research and education community.

Other research, educational, and government communities face application and communications needs very similar to those within higher education. These needs have spawned several high-performance networking initiatives that variously parallel and overlap Internet2, especially the NSF vBNS project and the set of "Next Generation Internet" efforts announced by the White House last fall. We fully intend that Internet2 develop in concert with these other efforts, thereby gaining synergy, minimizing duplication, and maximizing compatibility and interoperability among the resulting networks and applications.

The set of application areas listed above implies a corresponding set of communications requirements for Internet2. Future applications may require additional types of services. It is important that the Internet2 design be flexible enough to accommodate both currently anticipated requirements as well as new requirements as they become known.

Fundamental to the Internet2 infrastructure design is maintenance of a "common bearer service" for communication among network applications. The "bearer service" is the basic information transport interface for wide area communications, analogous to layer 3 in the ISO network model. One of the greatest strengths of the existing Internet is the ability of any node to communicate with any other node in a compatible transport format. We must preserve this strength in Internet2.

To the extent possible, the Internet2 bearer service must be backwards compatible with the existing commodity Internet. That existing infrastructure will continue to be the access path to all non-participants in Internet2, as well as to university members served by local Internet Service Providers (ISPs).

The common bearer service today is the Internet Protocol (IP) version 4. Internet2 will deploy IP version 6 (IPv6) as early as possible. All implementations must be backward compatible with IPv4.

In addition to IPv6, Internet2 must enable applications to specify a network "quality of service" (QoS) along dimensions including transmission speed, bounded delay and delay variance, throughput, and schedule. Meeting these requirements as soon as possible is the design challenge we have undertaken. Fortunately technologies intended to provide these capabilities have been under development for several years, and initial production versions are almost ready for serious testing in the field.

Beyond the technologies themselves, Internet2 participants will require both the most cost-effective services and services with predictable costs. Advanced services delivery and support systems will not be inexpensive. As these services migrate into the commodity market, we want to ensure that there is a sound economic model for their use. The engineering design for the Internet2 Project infrastructure must enable end-user management of costs as well as services.

1.B. Engineering Overview

A number of technical and practical considerations underlie the overall architecture for Internet2 infrastructure. One of these is the need to minimize overall costs to the participating campuses by providing for access to both the commodity Internet and advanced services through the same high-capacity local connection circuit. In addition, other campus programs and projects can be accommodated by means of a flexible regional interconnection architecture. For example, a metropolitan area network service might offer high capacity Internet service to student and faculty residences, and the campus will want a high capacity interconnection with that service.

For wide-area advanced services, a single interconnect service among gigaPoPs (probably the NSF-sponsored vBNS) will suffice at first. A number of service providers will be able to offer attractive advanced services as technologies migrate into the private sector. The design of Internet2 must optimize the campus's ability to acquire services from the widest variety of service providers.

The overall architecture of Internet2 is shown in Figure 1. The key new element in this architecture is the gigaPoP (for "gigabit capacity point of presence") -- a high-capacity, state-of-the-art interconnection point where Internet2 participants may exchange advanced services traffic with other Internet2 participants. Campuses in a geographic region will join together to acquire a variety of Internet services at a regional "gigaPoP".

Each campus (such as Alpha and Baker in Figure 1) will install a high speed circuit to its chosen gigaPoP through which it will gain access to commodity Internet services as well as advanced Internet2 services. The gigaPoPs will then join together to acquire and manage connectivity among themselves, in an organization whose structure and legal form remain to be determined but which we provisionally call the Collective Entity (CE). Potentially there could be a wide range of services available at the gigaPoP, limited only by the economics of the market and the absolute priority and insulation of Internet2 services.

To meet the requirements of Internet2 applications and developers, there must be advanced-services support both within the campus and among the gigaPoPs. Within the campus, there will be many ways to tackle this requirement, and we do not propose to enumerate them here. Among the gigaPoPs, the wide area interconnect service must support differentiated Quality of Service as well as highly reliable, high-capacity transport. Since these capabilities are not yet available in the commodity backbone Internet, a special purpose inter-gigaPoP interconnect network will be established by the Collective Entity. We expect that initially this interconnect will be provided by the NSF vBNS. Over time, however, vBNS connectivity will be augmented with other interconnection pathways to give Internet2 a redundant and comprehensive set of connections.

The gigaPoP concept can greatly increase market competition among Internet service providers and help ensure cost-effective Internet2 services over the long run. It might become a common way for end-user networks to acquire a wide variety of communications services, from basic Internet transport through caching and content provision.

Internet2 has four major technical components:

Applications that require Internet2-level services, such as those the Applications working group has outlined, and the equipment end users need to run the applications (denoted by solid-colored screens in Figure 1);

Campus networks connecting gigaPoPs to end users in their labs, classrooms, or offices (solid clouds);

GigaPoPs consolidating and managing traffic from campus networks (striped clouds); and finally

Internet2 interconnections among the gigaPoPs (dotted cloud).

Cutting across these components are the protocols for specifying and providing connectivity, especially connectivity with specific Qualify of Service (QoS) dimensions; the network-management tools, data, and organizations necessary to keep everything running, and the accounting and cost-allocation mechanisms required to negotiate reasonable, efficient, and productive distributions of costs across Internet2 members.

We expect that operators of some gigaPoPs will provide additional connectivity as well. For example, they may serve other networks and end users beyond those within the Internet2 gigaPoP consortium. But this must be done in such a way that it does not spill over into the Internet2 network "cloud". In fact we define the Internet2 gigaPoP as the connection node among Internet2 member campuses, other Internet2 gigaPoPs, and local networks that serve local Internet2 members, even if the Internet2 gigaPoP operator also provides other services to Internet2 members or others. We elaborate this distinction below,
in Section 3.

Most gigaPoPs will come into being through regional collaboration, often involving existing arrangements or state-level mechanisms, although some of them may be provided commercially. Most connections between campuses and gigaPoPs will be negotiated by the campus and/or the gigaPoP; most connections among gigaPoPs will be negotiated by the gigaPoPs themselves through the Collective Entity.

The overall scope of Internet2 applications requires next-generation network services on an end-to-end basis. This implies very significant upgrades to most campus networks. As we pointed out above, Internet2 members are severally responsible for bringing their campus networks up to Internet2 standards. Except to comment on specific requirements as they arise we assume this work is in good hands.  In the remainder of this document we focus on the remaining critical topics: gigaPoPs and their interconnecting network cloud.

1.C. Scope, Context, & Timeline

Internet2 must encourage the development and deployment of advanced real-time multimedia applications, and the network infrastructure and service differentiation needed to support them. Since Internet2 connectivity is limited to members only (a relatively small number of educational institutions), this effort is not a replacement for the commercial Internet. However, we expect that lessons learned here will influence the commercial Internet.

Internet2 will be a standards-based but pre-competitive production network, not a network research experiment. Furthermore, a key guiding principle is to use off-the-shelf technology wherever possible. Nevertheless, in implementing Internet2 we must pursue some research questions, and these distinguish Internet2 from a commodity service procurement. The research questions related to the network itself (as opposed to specific application areas) include:

Network service requirements. In particular, what network QoS levels are really needed for advanced real-time multimedia applications?

Protocols for delivering different QoS levels. In particular, how much state information must be maintained in routers and/or switches to deliver high-quality differentiated service? Is it possible to achieve the levels of QoS support we want without using link-level circuit switching?

Management. What are the administrative implications of a multi-QoS network, especially from network-management and cost-allocation perspectives?

Cost Recovery. How can authorization and attribution for QoS requests be handled efficiently in a "stateless" communications service?

The bulk of investment in Internet2 will go to procuring commercial data transport services and switching/routing equipment from the private sector, but we must also devote resources to answering these questions.

Internet2 obviously functions in the larger context of other significant national networks. For important example, mission networks run by other federal agencies, such as NASA and Energy, are important elements of the research networking environment. Agency needs for measurements and statistics can be coupled with Internet2 developments. And clearly agency initiatives such as the recent Magnetic Fusion Collaboratory are related to the application goals of Internet2. Internet2 seeks to complement and enhance other large-scale, high-performance networking initiatives, and to collaborate, cooperate, and join with them wherever possible. We hope and expect that it will no longer be necessary for each Federal project to pay for a separate direct connection into campus labs. Instead, Federal backbones can connect to gigaPoPs and utilize or build on the campus-gigaPoP linkage.

There are several ways to implement this symbiosis. Coordination is key. The federal labs have critical importance to the Internet2 membership. The federal networks provide connectivity for key academic experiments. Coordination should occur in engineering, routing, statistics, and diagnostics and trouble tickets. A regular forum should be established between Internet2 and federal networking leaders to exchange information and mutually plan advanced networking activities.

A few prototype gigaPoPs already exist. In some respects, for example, the Chicago-area Metropolitan Research and Education Network is a partial model for a gigaPoP. Similar projects in North Carolina and in California are near fruition. We expect these and a few other early aggregation points to be capable of handling their owners' Internet2 traffic quite quickly, certainly within the next six months or so. We also expect these gigaPoPs to be able to communicate with each other during this time frame, but with high bandwidth probably preceding QoS capabilities.

We also expect the number of Internet2 members whose campus networks provide Internet2 capability to grow significantly during the next six months. Over the following 12 months we expect most Internet2 members to upgrade at least their campus backbones, most gigaPoPs to become operational, and connections among Internet2 members and gigaPoPs to be working. Within about 24 months of project inception we expect teachers and researchers on Internet2 campuses to have routine access to Internet2 network service for their network-based applications.

1.D. Supporting and Assisting End Users

We do not expect gigaPoP operators to provide direct help desk or similar services for end users. Those users whose applications require Internet2 services will need to find help via their campus IT support organizations. The campus-level information-technology organization and the campus Network Operations Center (NOC) support staff will be able to participate in Internet2 technology forums sponsored by Internet2. We anticipate the usual electronic mail lists and on-line resources will be supported by the Collective Entity. Every discrete regional, state or national Internet2 network must have an associated NOC. This NOC will interact with other network NOCs and campus NOCs to diagnose and resolve problems. End users will be expected to report problems to their local NOC.

1.E. Principles

Six basic principles emerged in our deliberations, and are worth repeating here before we proceed to more technical material:

Buy rather than build. Wherever possible, we should employ currently available technology fully supported by its provider and used widely enough to have a track record. Failing this, we should seek technology likely to meet these criteria soon on the basis of provider commitment.

Open rather than closed. We should rely on open, published protocols and standards, avoid proprietary solutions, and promote full access to data on network performance.

Redundancy rather than reliance. We should try to avoid long-term dependence on single network providers, single hardware or software manufacturers, or single pathways.

Basics before complexity. Most members were drawn to Internet2 by specific needs, current or anticipated. We make sure that other possibilities do not distract us from meeting our needs.

Production not experimentation. Our goal is to provide support for the development of advanced applications by our members, not to become a laboratory for network experimentation. Of course some level of experimentation is necessary to keep improving the quality of production and to introduce new functions, and as we point out below gigaPoPs may provide this in parallel with production services, but it must not interfere with achievement of Internet2's key goals.

Services to end users, not among commercial providers. Internet2 gigaPoP functionality does not include peering among commercial networks. If a single organization or entity wishes to provide both Internet2 gigaPoP services and other, more commercial services, it must ensure that the two remain appropriately distinct.

2. Gigapops

2.A. Structure & Services

Logically, a gigaPoP is a regional network interconnect point providing access to the inter-gigaPoP network for (typically) several Internet2 members.

Organizationally, gigaPoPs are expected to be implemented by one or more universities, although there may be exceptions. For example, the Collective Entity might be called upon to operate certain gigaPoPs, universities might operate others on behalf of themselves and neighboring institutions, and others might be operated by a commercial entities.

It is neither practical nor possible to assign a single entity to run all gigaPoPs. Gigapop operation and coordination will be achieved through a an umbrella organization, which we have simply called the Collective Entity pending further structural discussions. Precedent for this mode of operation was set in the early Internet. It continues today in the North American Network Operators Group (NANOG). The Collective Entity must decide on common standards for interconnecting gigaPoPs and for the management protocols that will be exchanged in support of advanced communications services. This will include such arcane things as utilization measures that will support cost allocation, as well as research data that will enable characterization of the entire system.

Physically, a gigaPoP is a secure and environmentally conditioned location that houses a collection of communications equipment and support hardware. Circuits terminate there both from Internet2 members' networks and from wide-area data-transport networks, both Internet2 and commercial. Internet2 members' networks are assumed to be non-transit networks, that is, they don't carry traffic between a gigaPoP and the general Internet. Gigapops will serve end-user non-transit networks through appropriate IP route management. Internet2 gigaPoPs will not serve commercial transit networks, nor is peering allowed among such networks via the gigaPoP routing infrastructure. Inter-gigaPoP links will ONLY carry traffic among Internet2 sites.

A gigaPoP's key function is the exchange of Internet2 traffic with specified bandwidth and other Quality of Service attributes. In addition, standard IP traffic can be exchanged with commodity Internet service providers that have a termination at the gigaPoP in order to eliminate the need for separate high speed connections between the participant's campus network and other ISP exchange points. In many cases gigaPoPs will serve customers and purposes beyond communication among Internet2 applications developers. In particular, gigaPoPs may link Internet2 campus networks to:

• other metropolitan area networks in their communities, for example to provide local distance education;
• research partners and other organizations with which Internet2 members wish to communicate;
• other dedicated high-performance wide area networks, for example those that the government implements for its own research units; and
• other network services, for example commodity Internet backbone providers.

Gigapops will operate with minimal on-site staff. Operational support will be provided by a small number of Internet2 Network Operations Centers. However, no end-user support services will be available (see below).

Gigapops must participate in Internet2 operational management as well by collecting data on utilization, and sharing with one another and with campus-network operators the information necessary to schedule, provide, monitor, troubleshoot, and account for Internet2 network service.

We anticipate that each gigaPoP could serve five to ten Internet2 members. With even distribution this implies about a dozen gigaPoPs, but we think the number is unlikely to remain this low. First, geography will strongly influence gigaPoP affiliations, and Internet2 members do not distribute geographically in six-node clumps. Second, in many cases state-wide or regional initiatives will yield gigaPoPs serving both Internet2 and other needs (about which more below), and these are likely to be numerous. Third, some members for various reasons will implement their own gigaPoP, increasing numbers still further.

Can an Internet2 gigaPoP also provide similar services to non-Internet2 members, and perhaps even commercially? We have discussed this point at some length. Our conclusion is that the entity that provides connectivity among Internet2 members is an Internet2 gigaPoP if and only if it meets the functional and operational conditions we specify below, and does so without intentionally or accidentally providing Internet2 services -- especially inter-gigaPoP routing and transport -- to non-Internet2 applications or users.

The latter issue is joined whenever an Internet2 gigaPoP is part of some larger entity, perhaps merely a building also housing other connectivity equipment but perhaps a very sophisticated switching system capable of internally sorting Internet2 and non-Internet2 traffic without mishap. Mostly this is a matter of terminology. We can classify the things a gigaPoP might do into three categories:

1. The minimum a gigaPoP must do for Internet2 -- that is, the things it must do to satisfy the functional and operational specifications we outline below.
2. Things that a gigaPoP must not do on Internet2 -- for example, it must not route non-Internet2 traffic over Internet2 inter-gigaPoP connections, let other activities adversely affect Internet2 minimum performance, and so on.
3. All the other things a gigaPoP might do that are outside of Internet2.

An Internet2 gigaPoP, whatever its sponsorship and structure, must do the minimum things, must not do the forbidden things, and otherwise may operate as simply or otherwise as it chooses.

In practice we expect gigaPoPs to divide into two broad types:

• Type I gigaPoPs, which are relatively simple, serve only Internet2 members, route their Internet2 traffic through a one or two connections to another gigaPoPs, and therefore have little need for complex internal routing and firewalling; and
• Type II gigaPoPs, which are relatively complex, serve both Internet2 members and other networks to which Internet2 members need access, have a rich set of connections to other gigaPoPs, and therefore must provide mechanisms to route traffic correctly and prevent unauthorized or improper use of Internet2 connectivity.

We sketch all this in Figure 2. A Type I gigaPoP would omit some of the connections in Figure 2; in particular, the connections on the right side of the diagram would be confined to one or two connections to other gigaPoPs, perhaps one or two to local ISPs of importance to local Internet2 members, and perhaps one to a commodity Internet carrier.

We specify these different types because we believe that some aggregations of members will involve complex situations with significant traffic bound for and coming from diverse locations elsewhere, while others will involve relatively simpler and smaller aggregations whose drainage needs are more modest. What serves the former need will be overkill for the latter; what suffices for the latter will collapse faced with the demands of the former.

Whether the types are distinct tiers, or a continuum of some sort, will emerge as Internet2 is implemented and particularly as members associate into gigaPoPs. Given the rapidly growing numbers of Internet2 members and prospective gigaPoP consortia, it may be necessary to have some core switching nodes whose sole function is to connect gigaPoPs to one another. Since conceptually these are part of the network "cloud" interconnecting gigaPoPs, we will consider them only in that context.

External connections to gigaPoP ATM Switching Elements may be direct SONET circuits from campus ATM switches or other gigaPoP locations, or be full ATM service from commercial carriers. ATM Switching Elements serve to multiplex the link level bandwidth through permanent or switched virtual circuits (PVCs or SVCs). In this way, the intra- and inter-gigaPoP connectivity can be optimized and separate bandwidth can be allocated to testbed or other special purpose requirements.

The primary gigaPoP service is provided by the IP Routing Elements. These can be fed directly from external SONET/PPP or high speed synchronous circuits, or via PVC/SVC links into the ATM fabric. All QoS support and IP routing decisions are made in the equipment that does IP packet forwarding, and utilization data is extracted there. As technology permits, the IP packet forwarding equipment will be able to make use of the ATM layer to establish dynamic QoS or SVCs in support of differentiated IP service requirements.

2.B. Functional Requirements

An Internet2 gigaPoP's key function is to exchange traffic with specified bandwidth and other Quality of Service attributes between connected Internet2 member networks and the core Internet2 network. To achieve this goal, a gigaPoP must satisfy a variety of specific functional requirements.

2.B.1. Protocols

Since the Common Bearer Service of Internet2 is IP, any layer-3 devices in gigaPoPs will obviously support IP. But which IP? IPv4 is the current standard, but the Internet2 project can help move the community to IPv6. Thus, all Gigapop layer-3 devices should support IPv6 in addition to IPv4, as soon as stable implementations are available.

Of course, IP is not the only protocol in the TCP/IP suite. All the usual supporting protocols are assumed to be available where needed. In addition, IGMP (supporting multicast), and RSVP (supporting resource reservations) are expected to be very important to this project, and therefore should be available in all relevant gigaPoP devices.

2.B.2. Routing

The gigaPoPs are responsible for implementing any usage policies pertaining to Internet2. For example, to the extent that the vBNS is used to provide inter-gigaPoP connectivity, gigaPoPs must send only traffic destined for other Internet2 sites to their vBNS connection. Note that physical connectivity to a gigaPoP does not imply either permission or ability to exchange traffic with any other entity having a connection to that gigaPoP. Routing policies of the gigaPoP will be used to enforce not only the Internet2 rules, but also the bilateral peering agreements that will control local traffic exchange at gigaPoPs. We return to routing issues in Section 3.

2.B.3. Speed

The bit rate of connections into a gigaPoP or between gigaPoPs will vary widely, depending on the number and intensity of the Internet2-based applications running on its member campuses. The issue for the gigaPoP itself is to make sure that it has adequate capacity to handle the anticipated traffic load. The switches providing the primary interconnectivity in a gigaPoP, and the links from those switches to adjacent gigaPoP routers should be sized so that packet loss within the gigaPoP is near zero.

2.B.4. Linkage

Initial layer-2 connectivity to other gigaPoPs is expected to utilize ATM PVCs from the vBNS plus some dedicated links that may be ATM PVCs or SVCs, or raw SONET links. The linkages among gigaPoP routers connected to wide-area links will typically be provided by high-performance switches, typically either a cell-based or a frame-based service, depending on the needs of each specific gigaPoP.

2.B.5. Use Measurement

Costs for inter-gigaPoP connectivity are not yet known, and other gigaPoP costs will vary with circumstances and services offered, so it is not possible to say very much about requirements for cost accounting. Obviously, whatever pricing mechanisms are selected must be technically supportable. Gigapops must therefore save and share the usage statistics necessary for reasonable Internet2 cost allocations across members.

2.B.6. Facilitating regional aggregation

Gigapops are by definition aggregation points, but in some areas, digital transport costs may encourage a hierarchy of aggregation and exchange points within a region. In such cases, the Collective Entity may be able to play a constructive role in coordinating cost-effective regional Internet2 connectivity for affiliated institutions. A key goal of brokering such lower-level exchange points is to maintain consistency throughout the entire Internet2 infrastructure, both respect to the technical capabilities and network management policies and procedures.

2.B.7. Technology transfer

Just as the entire Internet2 project has as one of its goals the transfer of next-generation Internet technology to the commodity Internet, gigaPoPs will play a key role in transferring technology to member institutions. Although details will vary from one area to the next, there is an important opportunity for gigaPoP operators to share information with member institutions on deployment and management of their emerging multicast and multi-QoS campus networks.

2.B.8. Collaborations among gigaPoPs

Although multi-QoS and multicast connectivity among all Internet2 members is an explicit and important goal of the project, not all Internet2 members will be involved in every advanced application experiment. Indeed, some of these experiments will involve institutions served by a single gigaPoP . However, a likely scenario will be for several gigaPoPs to collaborate on specific application experiments and other projects. For example, multiple gigaPoPs might work together with private enterprise to facilitate improved connectivity for asynchronous and distance learning from member institutions to their constituents homes, just as gigaPoPs may facilitate local traffic exchange among commodity Internet Service Providers in their region.

2.B.9. Who can connect

Determination of which institutions or other exchange/aggregation points may connect to a particular gigaPoP is up to the management of that gigaPoP . Determination of who can exchange traffic at gigaPoP will depend on bilateral peering agreements between connectees as well as local gigaPoP policy. However, only Internet2 members may exchange traffic via the Internet2 "core" network, which links the gigaPoPs together.

2.B.10. Other gigaPoP services

It is not unreasonable to envision that gigaPoPs might accommodate caching nodes or even content servers in support of participant's activities. Since operational data collection within the gigaPoPs is a basic requirement, large capacity disks may be assumed to be on site. Caching could prove very effective in reducing demand on wide area links for some types of services. Similarly, content located at the gigaPoP would be readily available to the attached Internet2 participants as well as to the wide area links.

As an optional service for some Internet2 participants, ATM or other link level capacity might be made available upon special arrangement with the gigaPoP operator(s). It is anticipated that some researchers will benefit from a wide area testbed of this sort. With appropriate safeguards, it could be provided without interfering with normal Internet2 production services.

2.B.11. Performance expectations

Although a key goal of the Internet2 project is to understand how a multi-QOS network behaves under congestion, the gigaPoP itself should not become a bottleneck in access to wide area communications services. The total bandwidth capacity required by each Internet2 participant will vary but is expected to range from fractional DS-3 to as high as OC-12 (622 Mbps). The gigaPoP internal design must be able to handle the aggregate throughput demanded by all local participants and wide area connections. Gigapops must be able to provide the aggregate bandwidth while serving a number of customers with special QoS requirements.

2.C Operational Responsibilities

It is critical that the Internet2 project have a focal point for overall operational management. As we said earlier, this will require some organization -- a Collective Entity -- through which gigaPoPs collaborate to acquire bandwidth and achieve their other goals, which certainly include network management. At the very least the CE will require a technical coordinator at the national level and a coordination council that meets regularly. How these and other CE elements are defined and organized will be an issue for Internet2 management.

One of the overall goals of Internet2 is to enable study of the behavior of this complex and dynamic system. Such studies will include characterization of the traffic flows, analysis of queuing behavior in an environment delivering differentiated services, monitoring of end-to-end Internet2 performance, and review of various cost allocation and cost recovery models in light of actual use of the system. A part of the gigaPoP architecture must be integrated data collection with appropriate safeguards but with enough detail and accuracy to support serious study and analysis.

Internet2 will provide end-to-end dynamic services. This means that the end users can request particular network services between devices on Internet2 with the expectation that those services will be delivered regardless of the number of network providers involved in the traffic path. Various levels of service will be available and multiple connections at different levels of service may be requested at any time. An end user may not always get services requested if resources are not available to provide the level of service. However, once a request is granted the level of service will be guaranteed.

Because of the end-to-end nature of Internet2, operation of the network will require more coordination between network operators and between network operators and end users in most parts of the Internet. This coordination should be automated to the greatest extent possible. The current Internet lacks the tools and protocols to manage multiple levels of service. One Internet2 objective will be to work with standards bodies and developers to create these protocols and tools. In the development of these protocols and tools we must keep in mind that they will eventually be used in the commercial Internet, which operates in a different trust and sharing environment than the academic community.

The management of Internet2 services from gigaPoPs needs to be examined from two viewpoints. The first view is from the end user requesting the services, and the second from the network systems involved in providing those services. Within each of those viewpoints normal operations and problem resolution need to be considered.

2.C.1. Network management

The end-user request for service will be issued through the use of an application. The application will be responsible for interacting with the end user in the selection of service levels and advising of the availability and cost of the service. The application will also be responsible for interaction with the network system to obtain the services. How the applications, operating system, and network interface work together will be dependent on the implementation on the platform. We seek a difficult goal: uniform end-user presentation. Ideally, for example, error messages will be standardized so that the end user will understand the error even if the end user system is foreign to him or her, much as everyone understands a busy signal on a telephone, and will know what corrective action to take.

Management of the network system that provides Internet2 services may involve one or more networks operated by different entities. The network needs to work as a single system from the viewpoint of the end user. This requires that independently run networks coordinate service requests. Authentication and authorization for the use of resources is needed before the service request can be granted. Next, the system must determine whether the resources are available to meet the request and, if necessary, reserve them. Once the service request is granted, data on network resources consumed needs to be collected for appropriate resource control or cost accounting. To make the end-to-end service work, each network in the path must perform these steps in a coordinated fashion.

Current tools for network monitoring and diagnosis look at the network as individual devices and communications links. Typically this is just up/down status and some simple load information. These tools do not look at the network system as a whole or consider end-to-end performance. Tools need to be developed that take into consideration the end-to-end issues of various levels of service across multiple networks. Likewise, procedures for the human operators of different networks should be established to facilitate planning and problem resolution.

2.C.2. Service-level monitoring and data

The current Internet has one level of service, "best effort". In this environment it is easy to treat all end users as equals, or apportion costs based on non-dynamic parameters such as the bandwidth of the connection. When multiple levels of service are available some form of resource control or cost accounting must be implemented with feedback to the end user to ensure that the appropriate level of service is requested.

Since the best charging model for Internet2 is not obvious, Internet2 will be used initially to develop and test methods of cost allocation. Some goals are clear:

• The cost for a service should be predictable.
• Higher levels of service should cost more than lower levels.
• Accounting should be as simple as possible to minimize resources consumed by accounting.

Until appropriate models are developed, initial funding for Internet2 will have to follow the more traditional Internet models, such as equal sharing of costs perhaps pricing by connection speed.

2.C.3. Security

Clearly there is security that can be provided at the network layer and there is security that simply can not be achieved without greatly degrading other services.

Security concerns for Internet2 fall into three categories:

Network System Attacks. These are attacks on the network infrastructure itself where a person takes actions with intent to degrade or cause failure in the network system. These attacks can range from flooding the network, to spoofing network control protocols, to unauthorized access to network management systems. The result is denial of service to legitimate users of the network.

Unauthorized use of the network. As Internet2 provides different levels of service and different resource controls or fees are associated with these levels, network operators must protect against attempts to avoid these controls. Appropriate authentication and authorization are needed to obtain services. The methods and infrastructure for performing the authentication and authorization need to be secure against attack. This includes traditional security concerns such as not sending passwords in the clear to not being susceptible to replay or reuse of legitimate access information by unauthorized users.

Inappropriate use of the network. These are incidents that do not affect the network itself, but cause problems for end systems or people that use the network. They can include breaking into computer systems, theft of objects available over the network, harassment and many other crimes and policy violations. While the prevention, detection, and prosecution of these actions is outside the responsibility of the network operators, operators should be willing and able to aid investigation by appropriate authorities.

Network operators need to maintain their knowledge of traditional and new methods of attack in all categories. They also need to understand what measures can be used to detect and repel these attacks. Close coordination with other network operators as well as with organizations such as the CERT is required. Network operators should be able to provide reference to information on good operating procedures, etiquette in network use, and problem resolution to end network system operators.

The Internet strategy has always been that end systems are responsible for security of the applications. However, protocols and tools have evolved only slowly to fulfill this strategy. This has resulted in security measures, such as firewalls, being taken at the network. Although network-layer security for applications is often restrictive, it may have to be employed in the Internet2 network to provide the required security for applications. The use of network-layer security for applications should be as close to the end system as possible, i.e. at the LAN or campus level.

3. Connectivity Specifications and Sources

The basic architecture we conceive for the Internet2 communications infrastructure is illustrated in Figure 3. The various network segments in this diagram fall into two rough categories: those that connect the end user's application to the campus's gigaPoP (some of which Figure 3 subsumes into the campus-network clouds), and those that interconnect gigaPoPs. Since the former are largely a campus responsibility, provided basic standards are met, we devote most of our attention to inter-gigaPoP connections and to the routing and other protocols applying to all Internet2 network segments.

3.A. Intra-Campus and Campus-to-Gigapop

Internet2 project goals cannot be achieved unless campus networks are upgraded to provide adequate support for advanced applications. This means having a campus network in which applications requiring high-bandwidth, low-latency, low-jitter, and/or multicast routing may flourish. We expect different campuses to make different decisions on how best to achieve this goal. Some may rely on cell-switching backbones, while others will favor frame-based ethernet solutions, perhaps in conjunction with simple prioritization schemes. Still others will pursue RSVP or other IP bandwidth reservation techniques. In virtually all cases Internet2 members will need to upgrade their networks, at significant expense; this generally will be the largest fraction of their Internet2 investment.

One of the important research objectives of Internet2 is understanding just how much service differentiation is enough. This won't be a simple determination. For example, those interested in replacing conventional telephone services with an integrated services network may have more severe requirements than those focused on desktop collaboration tools.

We assume that most Internet2 campuses will require only high capacity circuits to the nearest gigaPoP and advanced-functionality routers as their campus gateways. Those sites wishing to support additional or experimental services also might install an ATM multiplexer or switch between the gigaPoP connection circuit and the campus border. Usually campus-to-gigaPoP connections will carry less traffic (average and peak) than inter-gigaPoP connections, and they may carry non-Internet2 traffic as well. In some cases there is no commercially available or financially feasible way to reach Internet2 campus-to-gigaPoP connection quality levels yet, and in these cases Internet2 bandwidth and selectable QoS may be unavailable on a member campus until the problem is resolved. Internet2 members also have committed to support border-to-end-user Internet2 service within their campuses. We expect that Internet2 connectivity will be available:

• very soon in a few locations on each member campus convenient to faculty and others who need it, and
• within 18 to 24 months throughout the campus, but perhaps not from closet or spine to desktop except as needed.

3.B. Gigapop-to-Gigapop

The key requirements for network interconnections among gigaPoPs are that they provide:

• very high reliability,
• high capacity (bandwidth),
• support for selectable QoS, and
• data-collection and circuit-management tools that gigaPoP and Internet2 overseers will need in order to assess and direct communications.

The specifics of inter-gigaPoP connections will depend on the bandwidth, QoS, and routing specifications they must meet. For practical reasons we assume that the basic underlying wide area transport will be provided over SONET with ATM signaling.

While gigaPoPs will be required to provide certain IP services, they will be encouraged but not required to support other intercampus communications experiments. In particular, gigaPoPs may work with attached campuses to broker connections based upon other communication services, such as direct ATM. In addition to such lower-layer alternatives, gigaPoPs will be expected to implement multicast routing and data transport in support of MBONE and similar architectures.

We expect the initial form of connectivity among gigaPoPs to be the NSF vBNS network. Over time we expect this to be expanded and enhanced by other forms of inter-gigaPoP connectivity.

Other possible linkages include:

• national network "clouds", such as Sprint's or IBM's;
• a national network cloud created and operated by Internet2; and
• individual point-to-point links between cooperating gigaPoPs.

Much depends on how the vBNS project evolves. One of the motivations for considering gigaPoP connections to additional core network clouds is the need to explore the implications of multiple service providers on a multi-QoS network. However, this is also a reason to delay seeking multiple providers, because the multi-vendor QoS settlements problem is expected to difficult. In addition, the network-management capability we have outlined for Internet2 requires significant networking suppliers to provide significant amounts of operational data. Suppliers have been loath to provide this in the past, and negotiations to obtain it probably will be lengthy and complex.

Although it is likely that there will be some point-to-point links between gigaPoPs in order to satisfy specific bandwidth or service needs, at this time we don't expect to build and operate a full-mesh nation-wide national network for Internet2 using conventional dedicated circuits. Instead, we expect to see virtual circuits provided from a commercial network cloud or from the vBNS unless additional analysis shows the roll-your-own strategy to be more cost-effective or necessary to achieve technical objectives.

The Collective Entity we have mentioned several times is central to the design, acquisition, and operation of the Internet2 network cloud, whoever its suppliers are. We believe that the CE should take some corporate form, so that it can negotiate and enforce contracts effectively. Whether the CE should be incorporated for this narrow purpose or more broadly remains to be determined; this is a policy question for top-level resolution.

3.C. Routing and Quality-of-Service Protocols

In Internet2, internet layer routing will be managed for both IPv4 and IPv6.
The Internet2 gigaPoPs will be constructed by consortia of universities, and the consortia will have their own infrastructure to interconnect their gigaPoP(s) and their members. In many cases consortium gigaPoPs will provide consortium-specific gigaPoP services to members of their consortia before they are connected to other gigaPoPs. In particular the consortia may have already established routing policies for traffic within and between themselves, as well as between themselves and other network services, before they connect to each other. Unlike other networks where a backbone and all backbone switches are owned and managed together, Internet2 will be built by linking units under coordinated but separate administrations.

Past experience has shown that it is all too easy for one entity providing specialized services to its members to accidentally leak inappropriate routes to other entities. Routing information thus needs to be filtered, ideally, and routing between the gigaPoPs to be performed using an inter-domain routing protocol. This would both give gigaPoP consortia more freedom in their routing policies and provide mutual protection against accidental route leaking.

However, we also want to provide support for QoS-based routing. At the moment support for QoS in inter-domain routing is close to nonexistent. There is a tradeoff here between network functionality and network predictability. If close coordination between the gigaPoPs is possible, then we can attempt to use an intra-domain protocol. This sort of close coordination only will be possible at all if the number of participants in Internet2 remains small (where "small" is measured in terms of how close the coordination is, for example, where routing administrators exchange e-mail regularly on a first-name basis).

Since there is no routing protocol which satisfies both our needs, and it does not look like there will be one for several years, we need to find ways to engineer around the problem at the outset, and to promote research into routing for the longer term.

Discussion of what is possible for various protocol families follows. First we discuss the basics of routing, with or without QoS awareness, and then add discussion of the enhanced possibilities which will be important to Internet2. In all cases, the usefulness of various aspects of QoS routing can be investigated hand-in-hand with explorations of resource management, assigning value to network resources, and pricing.

3.C.1. Routing for IPv4

Internet2 will only be used by Internet2 members as a transit network to reach (1) other Internet2 participants, or (2) other special research networks (such as the vBNS or ESnet) through designated paths. A consortium may establish connections to the commercial Internet, and to other services, for its own purposes, but will not propagate any information received from them into Internet2. Routing information will be filtered strictly. Generally a gigaPoP will propagate routing information only for sites known to be participants in the Internet2 project. In addition a gigaPoP may propagate routing information for sites on another research network if the source of that routing information within Internet2 is a designated path to that site and/or network. Such designations will be decided by the Collective Entity.

Decisions on route propagation within a gigaPoP consortium are purely that consortium's business. We recommend that a consortium propagate to its members information about reachability to other Internet2 participants via Internet2, but only if the members can be relied on not to leak this information outside of the consortium. That is, a consortium, including all of its member universities, must not inadvertently tell sources outside of Internet2 to reach sites elsewhere on Internet2 through it, unless it is a designated path for interconnecting with those sources. Internet2 will not provide transit routing among other backbone networks.

QoS-capable routing protocols for IPv4 are still scarce, if they exist at all. There is no support for QoS in either BGP or IDRP. QoS-capable OSPF is still being worked out.

Integrated PNNI is a possibility. The intent of I-PNNI is to use the routing protocol developed for PNNI for both ATM and IP. PNNI has drawn upon the knowledge gained in using its predecessors and has advantages as a routing protocol design. I-PNNI is intended to offer QoS-based routing to IP as well as ATM. It is not an inter-domain protocol (yet -- the possibility is being looked into), but has abstraction and aggregation of network elements.

QoS-capable routing for IPv4 will be part of Internet2's development agenda. This does not mean that the Internet2 community will necessarily do the work, but rather that the Internet2 community will give priority to promoting the development of QoS-capable routing through various means.

3.C.2. Routing for IPv6

IPv6 routing is still under development. I-PNNI is intended to have support for IPv6. IDRP has support for IPv6 in theory, but IDRP implementations are not considered strategic and will need work. IDRP has limited QoS support. At this time it looks like IDRP will be superseded by a new project, BGP4++. OSPF and RIP for IPv6 have been tentatively specified, but QoS-capable OSPF is still being developed. Those sites wishing to experiment with IPv6 may use either RIPv6 or static routes until appropriate routing protocols have been implemented. This is feasible because we can expect few sites to be working with IPv6 in the near future, and tight coordination between them will be possible. The static routes will need to be implemented without regard for any hierarchy of relationships in the Internet2 project. QoS routing for IPv6 will be part of Internet2's development agenda.

IPv6 addresses can be allocated by the Collective Entity

3.C.3. Route information at the ATM layer

ATM route information will be necessary because many of the QoS-related network functions with which we wish to experiment involve dynamic allocation of resources at the ATM layer. ATM can be expected to use permanent virtual connections for some functions (for example, carrying IP packets which do not require special virtual connections) and switched virtual connections for others. Where possible, switched virtual connections are always preferable to permanent virtual connections, to minimize complexity of configuration and to support rerouting in case of network problems.

Intra-domain routing has been developed for ATM (PNNI). At this time there are no policy filters available in any commercial ATM product. However, ATM routing does have effective QoS support. Until more sophisticated routing is available, ATM routing will have no filtering. This is feasible because we can expect few sites to be working with ATM in the near future, and tight coordination between them will be possible. It is also feasible with less coordination than IP routing because virtual connection setup can be monitored and managed.

ATM addresses can be allocated by the Collective Entity

3.C.4. Routing and Internet2 Hierarchy

The above sections only deal with the topmost levels of Internet2, and treat the lower layers as opaque, although it makes recommendations for them. The general rules laid out above for interactions between gigaPoPs, as well as between gigaPoPs and external networks, apply at every level within Internet2.

3.C.5. Quality of Service Dimensions

Based on discussions thus far, and likely to change as concrete applications take form, we expect Internet2 to permit requests for at least five dimensions of Quality of Service (QoS):

• Transmission speed. The minimum effective data rate to be provided, plus perhaps a target average and a tolerable maximum limit. Thus, for example, a user might request a connection whose data rate never falls below 50Mbps, and agrees not to expect transmission faster than 100Mbps.
• Bounded delay and delay variance. Especially for video and other signals that carry real-time information, the maximum effective interruption allowed. A user might specify that there be no gap between packets long enough to interrupt or freeze live video.
• Throughput. The amount of data to be transmitted in a specified time period. A user might specify that a terabyte of data be moved within ten minutes.
• Schedule. The starting and ending times for the requested service. A user might specify that the requested connectivity be available at some exact time in the future for some specified period (which of course would arise from the other QoS specifications).
• Loss rate. The maximum packet loss rate to be expected within a specified time interval.

The more extreme the QoS requested, the more demanding it is of network resources, and the more disruptive a request is to other users. These costs of providing service must be clear enough to users that they are encouraged not to request any higher level of service than they need. Whether complete information and communal spirit are sufficient remains to be seen. We expect that universities will prefer predictable costs at an institutional level, but may offer different allocation schemes to users on campus. Indeed, part of the research agenda for Internet2 is to identify the economic and public policy issues that reflect both marketplace and social forces. It is likely that within campuses, several allocation schemes may be employed, including those that foster rational consumption and some that address other goals.

We expect that Internet2 traffic will involve IP routing over ATM switching over SONET transmission, but as outlined under "connectivity" it may be too early to resolve this. We expect that RSVP and related protocols will communicate QoS requests, and that management of links up and down the network hierarchy will serve those requests.

4. Action Items

Some of the work necessary to make Internet2 real is already being done, and will be complete about six months from the project's beginning -- that is, by the summer of 1997. A much larger fraction of the work will take an additional twelve months, bringing us to the eighteen-month milestone. Some work probably will take the further six months that remains within the two-year implementation period for Internet2. We group below some of the key foci for our work together over the next two years.

4.A. Campus

Many Internet2 members already have plans, and perhaps active projects, to upgrade their campus networks to Internet2 service levels. In general such upgrades begin with the campus backbone and a few sites on campus with special connections. We expect that all Internet2 member institutions will have network-upgrade projects underway by the six-month milestone, and that most of these projects will be complete -- at least for the backbone and a reasonably distributed array of sites on campus -- by the eighteen-month milestone. We expect Internet2-level connectivity to be available routinely throughout most Internet2 campuses by the two-year milestone.

The major immediate networking action items for Internet2 campuses are four:

1. Plan and implement the necessary upgrades to campus network backbones and tail circuits,
2. Collaborate with other nearby campuses to design, fund, and implement a common gigaPoP,
3. Arrange for connectivity between the campus and the gigaPoP, and
4. Provide support for users whose applications require Internet2 connectivity

4.B. GigaPoP

The key items here are seven:

1. Organize and staff the gigaPoP,
2. Identify and secure a location for the gigaPoP installation,
3. Develop a gigaPoP design in coordination with the CE and other GP operators,
4. Acquire, install, test GP equipment and routing design.
5. Connect and test pipes to Internet2 members, local ISPs, regional networks, and other gigaPoP participants,
6. Connect and test pipes to other gigaPoPs as part of the Internet2 cloud, and
7. Establish working relationships with campus and Collective Entity network operators.

4.C. Cloud

The key items here are three, and parallel closely those for gigaPoPs:

1. Organize and staff the Collective Entity,
2. Agree on what data and control should be available to Collective Entity network managers, and
3. Negotiate network connectivity for the Internet2 cloud, beginning with vBNS but looking forward to other providers as well.

4.D. Overall

Several additional action items fall to all of us involved with Internet2, acting together through our gigaPoP consortia and the umbrella Collective Entity, whatever form it may take. We assume that the current Internet2 Steering Committee will evolve with the project, perhaps spawning a larger Council representing all Internet2 actors to debate and implement goals, constraints, cost allocation, and policy. This organizational question is not ours to resolve, but the resulting group(s) must tackle these items:

1. Appoint an Engineering Task Force to develop detailed model implementations, emphasizing the principles we outlined in Section 1.
2. Seek industry partners on a broad scale to gain access to state-of-the-art critical equipment and communications services at the best possible cost. These partnerships may take many different forms, from volume discounts for all Internet2 participants to donations of equipment to particular subsets of Internet2 institutions in support of specific development projects.
3. Create a more specific implementation timeline and targets that can be used to measure progress.

Without a single focal point dedicated to our collective success in this venture, there inevitably will be confusion and dispersion of effort. Along with appropriate organizational creation and evolution, we urge the early appointment of an engineering director with strong Internet technical knowledge and significant stature in the community to lead this critical effort.

Internet2 can succeed only if each of the major actors -- the member campuses, the gigaPoP consortia they form, the national entity that arranges inter-gigaPoP connectivity, and the various government and commercial partners in the whole -- carries its weight at the appointed time.