您的位置:首页 > 编程语言 > Java开发

springcloud-sleuth源码解析1-初始化

2020-02-15 00:05 525 查看

基于spring cloud 1.2.1版本

spring cloud的基础是spring boot,而spring boot可以零配置初始化spring。当我们引入spring-cloud-starter-sleuth依赖的时候,会附带spring-cloud-sleuth-core依赖。spring boot扫描spring-cloud-sleuth-core依赖下的spring.factories文件,如下:

# Auto Configuration
org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
org.springframework.cloud.sleuth.autoconfig.TraceAutoConfiguration,\
org.springframework.cloud.sleuth.metric.TraceMetricsAutoConfiguration,\
org.springframework.cloud.sleuth.log.SleuthLogAutoConfiguration,\
org.springframework.cloud.sleuth.instrument.messaging.TraceSpanMessagingAutoConfiguration,\
org.springframework.cloud.sleuth.instrument.messaging.TraceSpringIntegrationAutoConfiguration,\
org.springframework.cloud.sleuth.instrument.messaging.websocket.TraceWebSocketAutoConfiguration,\
org.springframework.cloud.sleuth.instrument.async.AsyncCustomAutoConfiguration,\
org.springframework.cloud.sleuth.instrument.async.AsyncDefaultAutoConfiguration,\
org.springframework.cloud.sleuth.instrument.hystrix.SleuthHystrixAutoConfiguration,\
org.springframework.cloud.sleuth.instrument.scheduling.TraceSchedulingAutoConfiguration,\
org.springframework.cloud.sleuth.instrument.web.TraceHttpAutoConfiguration,\
org.springframework.cloud.sleuth.instrument.web.TraceWebAutoConfiguration,\
org.springframework.cloud.sleuth.instrument.web.client.TraceWebClientAutoConfiguration,\
org.springframework.cloud.sleuth.instrument.web.client.TraceWebAsyncClientAutoConfiguration,\
org.springframework.cloud.sleuth.instrument.web.client.feign.TraceFeignClientAutoConfiguration,\
org.springframework.cloud.sleuth.instrument.zuul.TraceZuulAutoConfiguration,\
org.springframework.cloud.sleuth.instrument.rxjava.RxJavaAutoConfiguration,\
org.springframework.cloud.sleuth.annotation.SleuthAnnotationAutoConfiguration

# Environment Post Processor
org.springframework.boot.env.EnvironmentPostProcessor=\
org.springframework.cloud.sleuth.autoconfig.TraceEnvironmentPostProcessor

这些是自动装配bean的javaConfig,下面首先分析TraceAutoConfiguration。

TraceAutoConfiguration

@Configuration
@ConditionalOnProperty(value="spring.sleuth.enabled", matchIfMissing=true)
@EnableConfigurationProperties({TraceKeys.class, SleuthProperties.class})
public class TraceAutoConfiguration {
@Autowired
SleuthProperties properties;

@Bean
@ConditionalOnMissingBean
public Random randomForSpanIds() {
return new Random();
}

@Bean
@ConditionalOnMissingBean
public Sampler defaultTraceSampler() {
return NeverSampler.INSTANCE;
}

@Bean
@ConditionalOnMissingBean(Tracer.class)
public DefaultTracer sleuthTracer(Sampler sampler, Random random,
SpanNamer spanNamer, SpanLogger spanLogger,
SpanReporter spanReporter, TraceKeys traceKeys) {
return new DefaultTracer(sampler, random, spanNamer, spanLogger,
spanReporter, this.properties.isTraceId128(), traceKeys);
}

@Bean
@ConditionalOnMissingBean
public SpanNamer spanNamer() {
return new DefaultSpanNamer();
}

@Bean
@ConditionalOnMissingBean
public SpanReporter defaultSpanReporter() {
return new NoOpSpanReporter();
}

@Bean
@ConditionalOnMissingBean
public SpanAdjuster defaultSpanAdjuster() {
return new NoOpSpanAdjuster();
}

}

@Configuration与@Bean结合起来使用可以以javaConfig方式生成bean,@Bean annotation下面的方法会生成一个bean,方法名为bean的name,而入参则是创建该bean依赖的的ref bean。
@ConditionalOnProperty(value="spring.sleuth.enabled", matchIfMissing=true)注解的意思是该类生成bean需要的条件,这里条件是当spring.sleuth.enabled为true时(没有设置默认为true即条件符合,也就是显式设置为false才算不符合条件),才会触发生成bean。spring.sleuth.enabled的值在SleuthProperties中设置的,它会读取配置文件中以spring.sleuth为前缀的信息并设置到field中,比如enabled等。

@ConfigurationProperties("spring.sleuth")
public class SleuthProperties {

private boolean enabled = true;
/** When true, generate 128-bit trace IDs instead of 64-bit ones. */
private boolean traceId128 = false;

public boolean isEnabled() {
return this.enabled;
}

public void setEnabled(boolean enabled) {
this.enabled = enabled;
}

public boolean isTraceId128() {
return this.traceId128;
}

public void setTraceId128(boolean traceId128) {
this.traceId128 = traceId128;
}
}

@EnableConfigurationProperties({TraceKeys.class, SleuthProperties.class})注解则是支持将@ConfigurationProperties注解的bean注册到本类当中,这里是将TraceKey、SleuthProperties对应的bean注册到本类当中。

下面分析每个@Bean annotaded method,首先解释下@ConditionalOnMissingBean的含义。它的意思很明显,就是当不存在该类型的bean时才会触发。如果在其他地方定义了此类型的bean,那就不会执行该方法产生bean了。

  1. randomForSpanIds
    生成一个类型为Random,名称为randomForSpanIds的bean,此bean作用是为后续生成span id而创建的一个id生成器。
  2. defaultTraceSampler
    生成一个默认的trace采集器bean,该采集器为NeverSampler.INSTANCE,意思就是永远不采集。
  3. sleuthTracer
    生成一个DefaultTracer类型的trace bean,并将依赖的一些bean注入。该bean的主要作用就是创建span以及其他操作。
  4. spanNamer
    span名称生成器,这里返回一个DefaultSpanNamer类型的bean,能够实现通过SpanName注解设置span name。
  5. defaultSpanReporter
    span的上报器,比如将span上报给zipkin。这里返回一个NoOpSpanReporter类型的bean,即不做上报操作。
  6. defaultSpanAdjuster
    span的调整器,在span上报之前可以做一些调整。这里返回一个NoOpSpanAdjuster类型的bean,即不做任何调整。

sleuth的基础config分析完后,下面分析instrumnet的配置,即如何在应用中织入trace埋点的配置。

instrumnet

正常用到的trace span有三种类型:

  1. 作为server端接收请求、响应请求的span
  2. 作为client端发送请求、接收响应的span
  3. 作为本地模块代码块执行的span

1、2中的span属于remote span,而3中的span属于local span。

Web

好的,下面分析web的基础配置,也就是涉及到跨节点的trace配置。

TraceHttpAutoConfiguration

@Configuration
@ConditionalOnBean(Tracer.class)
@AutoConfigureAfter(TraceAutoConfiguration.class)
@EnableConfigurationProperties({ TraceKeys.class, SleuthWebProperties.class })
public class TraceHttpAutoConfiguration {

@Bean
@ConditionalOnMissingBean
public HttpTraceKeysInjector httpTraceKeysInjector(Tracer tracer, TraceKeys traceKeys) {
return new HttpTraceKeysInjector(tracer, traceKeys);
}

@Bean
@ConditionalOnMissingBean
public HttpSpanExtractor httpSpanExtractor(SleuthWebProperties sleuthWebProperties) {
return new ZipkinHttpSpanExtractor(Pattern.compile(sleuthWebProperties.getSkipPattern()));
}

@Bean
@ConditionalOnMissingBean
public HttpSpanInjector httpSpanInjector() {
return new ZipkinHttpSpanInjector();
}
}

@ConditionalOnBean(Tracer.class)意思是当存在Trace bean的时候才会触发。
@AutoConfigureAfter(TraceAutoConfiguration.class)意思是在TraceAutoConfiguration处理完后再触发。
@EnableConfigurationProperties({ TraceKeys.class, SleuthWebProperties.class })意思是将TraceKey、SleuthWebProperties对应的config注入到该类中,以便后期直接使用这些java config对象。

  1. httpTraceKeysInjector
    该bean主要功能是把http请求相关信息(比如request path、url、method等)添加到span中。
  2. httpSpanExtractor
    从http请求中提取trace信息(比如traceId,parent spanId等),根据提取的信息生成span,如果http请求中不存在trace信息,则会生成一个新的span。生成的span可能作为一个rpc请求的parent span。比如服务接受一个http请求,生成一个span a,然后内部通过rpc调用其他服务,这时候生成一个新的span b。span b的parent span就是 span a。
  3. httpSpanInjector
    该bean的作用是在client sent的时候也就是客户端发送请求的时候在request中(比如http request的head中)注入trace的相关信息,将相关信息传递到客户端请求的服务端。

TraceWebAutoConfiguration

前面分析的TraceHttpAutoConfiguration生成的bean只是一些核心基础的bean,但没有具体应用到web服务当中。而此类会进行相关配置。

@Configuration
@ConditionalOnProperty(value = "spring.sleuth.web.enabled", matchIfMissing = true)
@ConditionalOnWebApplication
@ConditionalOnBean(Tracer.class)
@AutoConfigureAfter(TraceHttpAutoConfiguration.class)
public class TraceWebAutoConfiguration {

/**
* Nested config that configures Web MVC if it's present (without adding a runtime
* dependency to it)
*/
@Configuration
@ConditionalOnClass(WebMvcConfigurerAdapter.class)
@Import(TraceWebMvcConfigurer.class)
protected static class TraceWebMvcAutoConfiguration {
}

@Bean
public TraceWebAspect traceWebAspect(Tracer tracer, TraceKeys traceKeys,
SpanNamer spanNamer) {
return new TraceWebAspect(tracer, spanNamer, traceKeys);
}

@Bean
@ConditionalOnClass(name = "org.springframework.data.rest.webmvc.support.DelegatingHandlerMapping")
public TraceSpringDataBeanPostProcessor traceSpringDataBeanPostProcessor(
BeanFactory beanFactory) {
return new TraceSpringDataBeanPostProcessor(beanFactory);
}

@Bean
public FilterRegistrationBean traceWebFilter(TraceFilter traceFilter) {
FilterRegistrationBean filterRegistrationBean = new FilterRegistrationBean(
traceFilter);
filterRegistrationBean.setDispatcherTypes(ASYNC, ERROR, FORWARD, INCLUDE,
REQUEST);
filterRegistrationBean.setOrder(TraceFilter.ORDER);
return filterRegistrationBean;
}

@Bean
public TraceFilter traceFilter(Tracer tracer, TraceKeys traceKeys,
SkipPatternProvider skipPatternProvider, SpanReporter spanReporter,
HttpSpanExtractor spanExtractor,
HttpTraceKeysInjector httpTraceKeysInjector) {
return new TraceFilter(tracer, traceKeys, skipPatternProvider.skipPattern(),
spanReporter, spanExtractor, httpTraceKeysInjector);
}

@Configuration
@ConditionalOnClass(ManagementServerProperties.class)
@ConditionalOnMissingBean(SkipPatternProvider.class)
@EnableConfigurationProperties(SleuthWebProperties.class)
protected static class SkipPatternProviderConfig {

@Bean
@ConditionalOnBean(ManagementServerProperties.class)
public SkipPatternProvider skipPatternForManagementServerProperties(
final ManagementServerProperties managementServerProperties,
final SleuthWebProperties sleuthWebProperties) {
return new SkipPatternProvider() {
@Override
public Pattern skipPattern() {
return getPatternForManagementServerProperties(
managementServerProperties,
sleuthWebProperties);
}
};
}

/**
* Sets or appends {@link ManagementServerProperties#getContextPath()} to the skip
* pattern. If neither is available then sets the default one
*/
static Pattern getPatternForManagementServerProperties(
ManagementServerProperties managementServerProperties,
SleuthWebProperties sleuthWebProperties) {
String skipPattern = sleuthWebProperties.getSkipPattern();
if (StringUtils.hasText(skipPattern)
&& StringUtils.hasText(managementServerProperties.getContextPath())) {
return Pattern.compile(skipPattern + "|"
+ managementServerProperties.getContextPath() + ".*");
}
else if (StringUtils.hasText(managementServerProperties.getContextPath())) {
return Pattern
.compile(managementServerProperties.getContextPath() + ".*");
}
return default
1aa70
SkipPattern(skipPattern);
}

@Bean
@ConditionalOnMissingBean(ManagementServerProperties.class)
public SkipPatternProvider defaultSkipPatternBeanIfManagementServerPropsArePresent(SleuthWebProperties sleuthWebProperties) {
return defaultSkipPatternProvider(sleuthWebProperties.getSkipPattern());
}
}

@Bean
@ConditionalOnMissingClass("org.springframework.boot.actuate.autoconfigure.ManagementServerProperties")
@ConditionalOnMissingBean(SkipPatternProvider.class)
public SkipPatternProvider defaultSkipPatternBean(SleuthWebProperties sleuthWebProperties) {
return defaultSkipPatternProvider(sleuthWebProperties.getSkipPattern());
}

private static SkipPatternProvider defaultSkipPatternProvider(
final String skipPattern) {
return new SkipPatternProvider() {
@Override
public Pattern skipPattern() {
return defaultSkipPattern(skipPattern);
}
};
}

private static Pattern defaultSkipPattern(String skipPattern) {
return StringUtils.hasText(skipPattern) ? Pattern.compile(skipPattern)
: Pattern.compile(SleuthWebProperties.DEFAULT_SKIP_PATTERN);
}

interface SkipPatternProvider {
Pattern skipPattern();
}
}

TraceWebAutoConfiguration执行的条件有这些:

  1. spring.sleuth.web.enabled不为false,不设置默认为true
  2. spring application context是一个web application context
  3. 当存在Trace bean
  4. 在TraceHttpAutoConfiguration初始化之后才能触发

内部静态类TraceWebMvcAutoConfiguration是为了import TraceWebMvcConfigurer的配置,它的条件是存在WebMvcConfigurerAdapter类,此类是spring mvc包中提供给用户自定义拦截器等功能的抽象类。
下面我们看下TraceWebMvcConfigurer:

@Configuration
class TraceWebMvcConfigurer extends WebMvcConfigurerAdapter {
@Autowired BeanFactory beanFactory;

@Bean
public TraceHandlerInterceptor traceHandlerInterceptor(BeanFactory beanFactory) {
return new TraceHandlerInterceptor(beanFactory);
}

@Override
public void addInterceptors(InterceptorRegistry registry) {
registry.addInterceptor(this.beanFactory.getBean(TraceHandlerInterceptor.class));
}
}

TraceWebMvcConfigurer定义了一个TraceHandlerInterceptor拦截器,然后重写addInterceptors方法,将拦截器设置到了spring mvc生命周期中。后期会对TraceHandlerInterceptor拦截器进行分析。
接下来分析TraceWebAutoConfiguration生成的每个bean。

traceWebAspect

@Aspect
public class TraceWebAspect {

private static final Log log = org.apache.commons.logging.LogFactory
.getLog(TraceWebAspect.class);

private final Tracer tracer;
private final SpanNamer spanNamer;
private final TraceKeys traceKeys;

public TraceWebAspect(Tracer tracer, SpanNamer spanNamer, TraceKeys traceKeys) {
this.tracer = tracer;
this.spanNamer = spanNamer;
this.traceKeys = traceKeys;
}

@Pointcut("@within(org.springframework.web.bind.annotation.RestController)")
private void anyRestControllerAnnotated() { }// NOSONAR

@Pointcut("@within(org.springframework.stereotype.Controller)")
private void anyControllerAnnotated() { } // NOSONAR

@Pointcut("execution(public java.util.concurrent.Callable *(..))")
private void anyPublicMethodReturningCallable() { } // NOSONAR

@Pointcut("(anyRestControllerAnnotated() || anyControllerAnnotated()) && anyPublicMethodReturningCallable()")
private void anyControllerOrRestControllerWithPublicAsyncMethod() { } // NOSONAR

@Pointcut("execution(public org.springframework.web.context.request.async.WebAsyncTask *(..))")
private void anyPublicMethodReturningWebAsyncTask() { } // NOSONAR

@Pointcut("execution(public * org.springframework.web.servlet.HandlerExceptionResolver.resolveException(..)) && args(request, response, handler, ex)")
private void anyHandlerExceptionResolver(HttpServletRequest request, HttpServletResponse response, Object handler, Exception ex) { } // NOSONAR

@Pointcut("(anyRestControllerAnnotated() || anyControllerAnnotated()) && anyPublicMethodReturningWebAsyncTask()")
private void anyControllerOrRestControllerWithPublicWebAsyncTaskMethod() { } // NOSONAR

@Around("anyControllerOrRestControllerWithPublicAsyncMethod()")
@SuppressWarnings("unchecked")
public Object wrapWithCorrelationId(ProceedingJoinPoint pjp) throws Throwable {
Callable<Object> callable = (Callable<Object>) pjp.proceed();
if (this.tracer.isTracing()) {
if (log.isDebugEnabled()) {
log.debug("Wrapping callable with span [" + this.tracer.getCurrentSpan() + "]");
}
return new SpanContinuingTraceCallable<>(this.tracer, this.traceKeys, this.spanNamer, callable);
}
else {
return callable;
}
}

@Around("anyControllerOrRestControllerWithPublicWebAsyncTaskMethod()")
public Object wrapWebAsyncTaskWithCorrelationId(ProceedingJoinPoint pjp) throws Throwable {
final WebAsyncTask<?> webAsyncTask = (WebAsyncTask<?>) pjp.proceed();
if (this.tracer.isTracing()) {
try {
if (log.isDebugEnabled()) {
log.debug("Wrapping callable with span [" + this.tracer.getCurrentSpan()
+ "]");
}
Field callableField = WebAsyncTask.class.getDeclaredField("callable");
callableField.setAccessible(true);
callableField.set(webAsyncTask, new SpanContinuingTraceCallable<>(this.tracer,
this.traceKeys, this.spanNamer, webAsyncTask.getCallable()));
} catch (NoSuchFieldException ex) {
log.warn("Cannot wrap webAsyncTask's callable with TraceCallable", ex);
}
}
return webAsyncTask;
}

@Around("anyHandlerExceptionResolver(request, response, handler, ex)")
public Object markRequestForSpanClosing(ProceedingJoinPoint pjp,
HttpServletRequest request, HttpServletResponse response, Object handler, Exception ex) throws Throwable {
Span currentSpan = this.tracer.getCurrentSpan();
try {
if (!currentSpan.tags().containsKey(Span.SPAN_ERROR_TAG_NAME)) {
this.tracer.addTag(Span.SPAN_ERROR_TAG_NAME, ExceptionUtils.getExceptionMessage(ex));
}
return pjp.proceed();
} finally {
if (log.isDebugEnabled()) {
log.debug("Marking span " + currentSpan + " for closure by Trace Filter");
}
request.setAttribute(TraceFilter.TRACE_CLOSE_SPAN_REQUEST_ATTR, true);
}
}

此bean对两种情况进行了aop拦截:

  1. 对RestController、Controller注解,然后这些条件组合进行Around处理。当在spring mvc的controller中有方法直接返回java.util.concurrent.Callable或者WebAsyncTask对象进行异步操作的时候是需要将trace的上下文设置到异步上下文中,方便后续trace的操作,所以需要对这些情况进行aop,包装trace信息,返回一个wrap对象。
  2. 对HandlerExceptionResolver.resolveException方法进行了拦截,针对异常做了一些标识处理。

traceSpringDataBeanPostProcessor

首先判断是否存在org.springframework.data.rest.webmvc.support.DelegatingHandlerMapping,即是否引入了spring data rest相关依赖包。没有的话则不会生成该bean。
该bean的主要作用是对Spring Data REST Controllers进行了包装,生成TraceDelegatingHandlerMapping代理类,该类对被代理类进行了拦截器设置,拦截器为前部分讲到的TraceHandlerInterceptor。

traceFilter

trace拦截器,对每个请求进行拦截,在spring mvc的拦截器之前触发。他的主要作用是从request head中获取trace信息,生成span,没有trace信息则生成一个新的span。

traceWebFilter

将traceFilter注册到servlet容器中。也就是用java config形式代替了在web.xml中配置filter。

spring sleuth的核心配置分析完后,下面分析本章最后的部分-ZipkinAutoConfiguration。

ZipkinAutoConfiguration

@Configuration
@EnableConfigurationProperties({ZipkinProperties.class, SamplerProperties.class})
@ConditionalOnProperty(value = "spring.zipkin.enabled", matchIfMissing = true)
@AutoConfigureBefore(TraceAutoConfiguration.class)
public class ZipkinAutoConfiguration {

@Bean
@ConditionalOnMissingBean
public ZipkinSpanReporter reporter(SpanMetricReporter spanMetricReporter, ZipkinProperties zipkin,
ZipkinRestTemplateCustomizer zipkinRestTemplateCustomizer) {
RestTemplate restTemplate = new RestTemplate();
zipkinRestTemplateCustomizer.customize(restTemplate);
return new HttpZipkinSpanReporter(restTemplate, zipkin.getBaseUrl(), zipkin.getFlushInterval(),
spanMetricReporter);
}

@Bean
@ConditionalOnMissingBean
public ZipkinRestTemplateCustomizer zipkinRestTemplateCustomizer(ZipkinProperties zipkinProperties) {
return new DefaultZipkinRestTemplateCustomizer(zipkinProperties);
}

@Bean
@ConditionalOnMissingBean
public Sampler defaultTraceSampler(SamplerProperties config) {
return new PercentageBasedSampler(config);
}

@Bean
public SpanReporter zipkinSpanListener(ZipkinSpanReporter reporter, EndpointLocator endpointLocator,
Environment environment, List<SpanAdjuster> spanAdjusters) {
return new ZipkinSpanListener(reporter, endpointLocator, environment, spanAdjusters);
}

@Configuration
@ConditionalOnMissingBean(EndpointLocator.class)
@ConditionalOnProperty(value = "spring.zipkin.locator.discovery.enabled", havingValue = "false", matchIfMissing = true)
protected static class DefaultEndpointLocatorConfiguration {

@Autowired(required=false)
private ServerProperties serverProperties;

@Autowired
private ZipkinProperties zipkinProperties;

@Autowired(required=false)
private InetUtils inetUtils;

@Value("${spring.application.name:unknown}")
private String appName;

@Bean
public EndpointLocator zipkinEndpointLocator() {
return new ServerPropertiesEndpointLocator(this.serverProperties, this.appName,
this.zipkinProperties, this.inetUtils);
}

}

@Configuration
@ConditionalOnClass(DiscoveryClient.class)
@ConditionalOnMissingBean(EndpointLocator.class)
@ConditionalOnProperty(value = "spring.zipkin.locator.discovery.enabled", havingValue = "true")
protected static class DiscoveryClientEndpointLocatorConfiguration {

@Autowired(required=false)
private ServerProperties serverProperties;

@Autowired
private ZipkinProperties zipkinProperties;

@Autowired(required=false)
private InetUtils inetUtils;

@Value("${spring.application.name:unknown}")
private String appName;

@Autowired(required=false)
private DiscoveryClient client;

@Bean
public EndpointLocator zipkinEndpointLocator() {
return new FallbackHavingEndpointLocator(discoveryClientEndpointLocator(),
new ServerPropertiesEndpointLocator(this.serverProperties, this.appName,
this.zipkinProperties, this.inetUtils));
}

private DiscoveryClientEndpointLocator discoveryClientEndpointLocator() {
if (this.client!=null) {
return new DiscoveryClientEndpointLocator(this.client, this.zipkinProperties);
}
return null;
}

}

}

ZipkinAutoConfiguration上的annotation就不一一说明,前部分已经涉及到了。下面直接分析生成的bean。

reporter

返回HttpZipkinSpanReporter类型bean,内部会初始化sender以及delegate。sender是以何种方式将zipkin span发送到zipkin server。delegate是一个委托类,内部创建一个有界队列,异步将zipkin span发送到zipkin server。

zipkinRestTemplateCustomizer

创建reporter bean时需要依赖的sender。这里zipkin使用restTemplate作为sender提交span。

defaultTraceSampler

默认采样器,之前在TraceAutoConfiguration中也声明创建defaultTraceSampler,但由于ZipkinAutoConfiguration上有这样的注解:@AutoConfigureBefore(TraceAutoConfiguration.class),意思是在TraceAutoConfiguration之前执行,所以默认采样器使用的是ZipkinAutoConfiguration中生成的采样器即PercentageBasedSampler,它可以设置百分比来进行采样。

zipkinSpanListener

sleuth event监听器,上报sleuth span,然后转换成zipkin span,再通过zipkin reporter上报到zipkin server。

本章就到此结束了,其实还有hystrix、async相关模块的配置没讲,本章只讲了web相关的,后期有机会再讲。

转载于:https://my.oschina.net/u/913896/blog/1143112

  • 点赞
  • 收藏
  • 分享
  • 文章举报
chenping0574 发布了0 篇原创文章 · 获赞 1 · 访问量 430 私信 关注
内容来自用户分享和网络整理,不保证内容的准确性,如有侵权内容,可联系管理员处理 点击这里给我发消息
标签: